2012 TrustworthyOnlineControlledExpe

(Kohavi et al., 2012) ⇒ Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. (2012). “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained.” In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2012). ISBN:978-1-4503-1462-6 doi:10.1145/2339530.2339653

Subject Headings:

Notes

Cited By

Quotes

Author Keywords

A/b testing; controlled experiments; experimental design; randomized experiments

Abstract

Online controlled experiments are often utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher's experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale -- thousands of experiments now -- has taught us many lessons. These exemplify the proverb that the difference between theory and practice is greater in practice than in theory. We present our learnings as they happened: puzzling outcomes of controlled experiments that we analyzed deeply to understand and explain. Each of these took multiple-person weeks to months to properly analyze and get to the often surprising root cause. The root causes behind these puzzling results are not isolated incidents; these issues generalized to multiple experiments. The heightened awareness should help readers increase the trustworthiness of the results coming out of controlled experiments. At Microsoft's Bing, it is not uncommon to see experiments that impact annual revenue by millions of dollars, thus getting trustworthy results is critical and investing in understanding anomalies has tremendous payoff: reversing a single incorrect decision based on the results of an experiment can fund a whole team of analysts. The topics we cover include: the OEC (Overall Evaluation Criterion), click tracking, effect trends, experiment length and power, and carryover effects.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2012 TrustworthyOnlineControlledExpe	Ron Kohavi Roger Longbotham Brian Frasca Toby Walker Alex Deng Ya Xu			Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained				10.1145/2339530.2339653		2012