Bivariate (A/B) Controlled-Experiment Test

A Bivariate (A/B) Controlled-Experiment Test is a two-treatment controlled experiment with two different active treatments (not a null experiment).

• AKA: Bucket Experiment, Split Bivariate Test, Randomized Experiment, Split Test, Online Controlled Experiment, Control/Treatment Test, Online Field Experiment.
• Context:
  • It can be created by an A/B Testing Task.
  • It can (often) be a Champion-Challenger Experiment.
  • …
• Example(s):
  • a Online Store Recommendation Strand A/B Test.
  • a Search Enginer A/B Test.
  • a Clinical Decision Support A/B Test.
  • …
• Counter-Example(s):
  • a Multivariate Experiment.
  • a Placebo Controlled Experiment.
  • an A/A Null Test.
• See: Student's t-Test.

---

---

References

2017

• (Kohavi & Longbotham, 2017) ⇒ Kohavi R., Longbotham R. (2017) Online Controlled Experiments and A/B Testing. In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA
  • ABSTRACT: The Internet connectivity of client software (e.g., apps running on phones and PCs), websites, and online services provide an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called A/B tests, split tests, randomized experiments, control / treatment tests, and online field experiments. Unlike most data mining techniques for finding correlational patterns, controlled experiments allow establishing a causal relationship with high probability. Experimenters can utilize the scientific method to form a hypothesis of the form “If a specific change is introduced, will it improve key metrics?” and evaluate it with real users.

    The theory of a controlled experiment dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, and the topic of offline experiments is well developed in Statistics (Box et al., Statistics for experimenters: design, innovation, and discovery. Wiley, Hoboken, 2005). Online-controlled experiments started to be used in the late 1990s with the growth of the Internet. Today, many large sites, including Amazon, Bing, Facebook, Google, LinkedIn, and Yahoo!, run thousands to tens of thousands of experiments each year testing user interface (UI) changes, enhancements to algorithms (search, ads, personalization, recommendation, etc.), changes to apps, content management system, etc. Online-controlled experiments are now considered an indispensable tool, and their use is growing for startups and smaller websites. Controlled experiments are especially useful in combination with Agile software development (Martin, Clean code: a handbook of Agile software craftsmanship. Prentice Hall, Upper Saddle River, 2008; Rubin, Essential scrum: a practical guide to the most popular Agile process. Addison-Wesley Professional, Upper Saddle River, 2012), Steve Blank’s Customer Development process (Blank, The four steps to the epiphany: successful strategies for products that win. Cafepress.com., 2005), and MVPs (minimum viable products) popularized by Eric Ries’s Lean Startup (Ries, The lean startup: how today’s entrepreneurs use continuous innovation to create radically successful businesses. Crown Business, New York, 2011).

2013

• (Wikipedia, 2011) ⇒ http://en.wikipedia.org/wiki/A/B_testing
  • QUOTE: A/B testing is a methodology of using randomized experiments with two variants, A and B, which are the Control and Treatment in the controlled experiment. Such experiments are commonly used in web development and marketing, as well as in more traditional forms of advertising. Other names include randomized controlled experiments, online controlled experiments, and split testing. In online settings, such as web design (especially user experience design), the goal is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. Version A might be the currently used version (Control), while Version B is modified in some respect (Treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significant improvements can be seen through testing elements like copy text, layouts, images and colors. Multivariate testing or bucket testing is similar to A/B testing, but tests more than two different versions at the same time.

    While the approach is identical to a between-subjects design, which is commonly used in a variety of research traditions, A/B testing is seen as a significant change in philosophy and business strategy in Silicon Valley. A/B testing as a philosophy of web development brings the field into line with a broader movement toward evidence-based practice.

2010

• http://promote.autonomy.com/components/pagenext.jsp?topic=TD::GLOSSARY_OF_TERMS
  • QUOTE: A/B Testing is the most simplistic way of conducting direct marketing and website tests. In such tests, the "A" option is the control, or current champion. The "B" option is the challenger being tested in an attempt to provide better results than "A." During a split run, visitors are randomly shown or offered the "A" or the "B" option. The difference between the two response rates is then evaluated for statistical significance. While simple to conduct and understand, A/B testing is much less informative and much more costly if more than two factors need to be tested, and has a much lower efficiency than multivariable experimental designs.

---