2009 SevenPitfallstoAvoidWhenRunning

From GM-RKB
Jump to navigation Jump to search

Subject Headings: Controlled Experiment Design, Online Optimization, A/B Testing.

Notes

Cited By

Quotes

Author Keywords

Controlled Experiments, A/B Testing, E-commerce, Simpson’s Paradox, Robot Detection

Abstract

Controlled experiments, also called randomized experiments and A/B tests, have had a profound influence on multiple fields, including medicine, agriculture, manufacturing, and advertising. While the theoretical aspects of offline controlled experiments have been well studied and documented, the practical aspects of running them in online settings, such as web sites and services, are still being developed. As the usage of controlled experiments grows in these online settings, it is becoming more important to understand the opportunities and pitfalls one might face when using them in practice. A survey of online controlled experiments and lessons learned were previously documented in Controlled Experiments on the Web: Survey and Practical Guide (Kohavi, et al., 2009). In this follow-on paper, we focus on pitfalls we have seen after running numerous experiments at Microsoft. The pitfalls include a wide range of topics, such as assuming that common statistical formulas used to calculate standard deviation and statistical power can be applied and ignoring robots in analysis (a problem unique to online settings). Online experiments allow for techniques like gradual ramp-up of treatments to avoid the possibility of exposing many customers to a bad (e.g., buggy) Treatment. With that ability, we discovered that it's easy to incorrectly identify the winning treatment because of Simpson's paradox.

References

  • 1. Bacher, Paul, Et Al. 2005. Know Your Enemy: Tracking Botnets. The Honeynet Project. {Online} March 13, 2005. Http://www.honeynet.org/papers/bots/.
  • 2. Bomhardt, Christian, Gaul, Wolfgang and Schmidt-Thieme, Lars. 2005. Web Robot Detection - Preprocessing Web Logfiles for Robot Detection. {book Auth.} Maurizio Vichi, Et Al. New Developments in Classification and Data Analysis. S.l. : Springer, 2005.
  • 3. Box, George E.P., Hunter, J Stuart and Hunter, William G. 2005. Statistics for Experimenters: Design, Innovation, and Discovery. 2nd. S.l. : John Wiley&Sons, Inc, 2005. 0471718130.
  • 4. Mark Claypool, David Brown, Phong Le, Makoto Waseda, Inferring User Interest, IEEE Internet Computing, v.5 n.6, p.32-39, November 2001 doi:10.1109/4236.968829
  • 5. Efron, Bradley and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. New York : Chapman&Hall, 1993. 0-412-04231-2.
  • 6. Fieller, E C. 1940. The Biological Standardization of Insulin. Supplement to the Journal of the Royal Statistical Society. 1940, Vol. 7, 1, Pp. 1--64.
  • 7. Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, Thomas White, Evaluating Implicit Measures to Improve Web Search, ACM Transactions on Information Systems (TOIS), v.23 n.2, p.147-168, April 2005 doi:10.1145/1059981.1059982
  • 8. Hill, Nigel, Roche, Greg and Allen, Rachel. (2007). Customer Satisfaction: The Customer Experience Through the Customer's Eyes. S.l. : Cogent Publishing, 2007.
  • 9. Hopkins, Claude. 1923. Scientific Advertising. New York City : Crown Publishers Inc., 1923.
  • 10. Keppel, Geoffrey, Saufley, William H and Tokunaga, Howard. 1992. Introduction to Design and Analysis. 2nd. S.l. : W.H. Freeman and Company, 1992.
  • 11. Ron Kohavi, Roger Longbotham, Dan Sommerfield, Randal M. Henne, Controlled Experiments on the Web: Survey and Practical Guide, Data Mining and Knowledge Discovery, v.18 n.1, p.140-181, February 2009 doi:10.1007/s10618-008-0114-1
  • 12. Ron Kohavi, Llew Mason, Rajesh Parekh, Zijian Zheng, Lessons and Challenges from Mining Retail E-Commerce Data, Machine Learning, v.57 n.1-2, p.83-113, October-November 2004 doi:10.1023/B:MACH.0000035473.11134.83
  • 13. Ron Kohavi, Randal M. Henne, Dan Sommerfield, Practical Guide to Controlled Experiments on the Web: Listen to Your Customers Not to the Hippo, Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 12-15, 2007, San Jose, California, USA doi:10.1145/1281192.1281295
  • 14. Koselka, Rita. 1996. The New Mantra: MVT. Forbes. March 11, 1996, Pp. 114--118.
  • 15. Malinas, Gary and Bigelow, John. 2004. Simpson's Paradox. Stanford Encyclopedia of Philosophy. {Online} 2004. {Cited: February 28, 2008.} Http://plato.stanford.edu/entries/paradox-simpson/.
  • 16. Mason, Robert L, Gunst, Richard F and Hess, James L. 1989. Statistical Design and Analysis of Experiments With Applications to Engineering and Science. S.l. : John Wiley&Sons, 1989. 047185364X .
  • 17. Douglas C. Montgomery, Design and Analysis of Experiments, John Wiley & Sons, 2006
  • 18. Rao, C. Radhakrishna. 1973. Linear Statistical Inference and Its Applications. 2nd. S.l. : John Wiley&Sons, Inc., 1973.
  • 19. Roy, Ranjit K. 2001. Design of Experiments Using the Taguchi Approach : 16 Steps to Product and Process Improvement. S.l. : John Wiley&Sons, Inc, 2001. 0-471-36101-1.
  • 20. Simpson, Edward H. 1951. The Interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society, Ser. B. 1951, Vol. 13, Pp. 238--241.
  • 21. Spears, Steven J. 2004. Learning to Lead at Toyota. Harvard Business Review. May 2004, Pp. 78--86.
  • 22. Pang-Ning Tan, Vipin Kumar, Discovery of Web Robot Sessions Based on their Navigational Patterns, Data Mining and Knowledge Discovery, v.6 n.1, p.9-35, January 2002 doi:10.1023/A:1013228602957
  • 23. Wikipedia: Botnet. (2008). Botnet. Wikipedia. {Online} 2008. {Cited: February 28, 2008.} Http://en.wikipedia.org/wiki/Botnet.
  • 24. Wikipedia: Internet Bot. (2008). Internet Bot. Wikipedia. {Online} 2008. {Cited: February 28, 2008.} Http://en.wikipedia.org/wiki/Internet_bot.
  • 25. Wikipedia: Simpson's Paradox. (2008). Simpson's Paradox. Wikipedia. {Online} 2008. {Cited: February 28, 2008.} Http://en.wikipedia.org/wiki/Simpson%27s_paradox.

,

 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2009 SevenPitfallstoAvoidWhenRunningRon Kohavi
Thomas Crook
Roger Longbotham
Brian Frasca		Seven Pitfalls to Avoid when Running Controlled Experiments on the WebKDD-2009 Proceedingshttp://exp-platform.com/Documents/2009-ExPpitfalls.pdf10.1145/1557019.15571392009
Retrieved from "http://www.gabormelli.com/RKB/index.php?title=2009_SevenPitfallstoAvoidWhenRunning&oldid=900999"