n-Active-Treatment (Multivariate) Controlled Experiment

An n-Active-Treatment (Multivariate) Controlled Experiment is a treatment-controlled experiment with more than two active treatments.

• Context:
  • It can be a Placebo-included Multivariate Controlled Experiment.
  • It can be a Multivariate Null Controlled Experiment.
  • …
• Counter-Example(s):
  • a 2-Active-Treatment (A/B) Controlled Experiment.
  • a Placebo-Controlled Experiment.
  • a Multi-Armed Bandit Test.
  • an Active-Treatment Change Observational Analysis.
• See: Multivariate Test.

---

---

References

2012

• http://en.wikipedia.org/wiki/Multivariate_testing
  • QUOTE: In statistics, multivariate testing or multi-variable testing is a technique for testing hypotheses on complex multi-variable systems, especially used in testing market perceptions.

2012

• http://en.wikipedia.org/wiki/Multivariate_testing#Design_of_experiments
  • Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:
    1. Discrete choice and what has mutated to become choice modeling is the complex technique that won Daniel McFadden the Nobel Prize in Economics in 2000. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methods nor Optimal design solve for.
    2. Optimal design involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website.^{[citation needed]} This allows one to find the optimal solution unencumbered by limitations.
    3. Taguchi methods: with multiple variations of content in multiple locations on a website, a large number of combinations need to be statistically tested and medium/low traffic websites can take some time to get a large enough sample of visitors to decide which content gives the best performance. For example, if 3 different images are to be tested in 3 locations, there are 27 combinations to test. Taguchi methods (namely Taguchi orthogonal arrays) can be used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements. Taguchi uses fractional factorial designs.

2012

• (Lempel et al., 2012) ⇒ Ronen Barenboim, Edward Bortnikov, Nadav Golbandi, Amit Kagian, Liran Katzir, Ronny Lempel, Hayim Makabee, Scott Roy, and Oren Somekh. (2012). “Hierarchical Composable Optimization of Web Pages.” In: Proceedings of the 21st international conference companion on World Wide Web. doi:10.1145/2187980.2187987

2007

• (Torbeck, 2007) ⇒ Lynn D. Torbeck. (2007). “Pharmaceutical and Medical Device Validation by Experimental Design." CRC Press. ISBN:1420055690
  • QUOTE: Controlled multivariate experiments are the most logical, the most scientific, and the most efficient way that scientists know to collect data. … These observational tools cannot find and describe cause-and-effect relationships directly. The only way to find these relationships is to conduct a multivariate controlled-experiment.

    In contrast to the observational approach, data collection in an controlled experiment is active; investigators take control of the environment and critical process parameters. By deliberate changes in key factors, the cause-and-effect relationships are forced to show themselves.

---