Post Hoc Analysis

A Post Hoc Analysis is an a posteriori (i.e. after-the-fact) hypothesis testing, multiple comparison procedure, or data analysis method.

• AKA: A Posteriori Analysis, Unplanned Comparison, Posthoc Test, Posthoc Procedure, Retrospective Analysis.
• Example(s)
  • Fisher's Least Significant Difference Test
  • Bonferroni Correction
  • Holm-Bonferroni Method.
  • Newman-Keuls Method.
  • Duncan's New Multiple Range Test.
  • Rodger's Method
  • Scheffe's Method.
  • Tukey's Test.
  • Dunnett's Test.
  • Sidak's Inequality.
  • Benjamini-Hochberg Procedure.
• Counter-Example(s)
  • Bayesian Analysis.
  • Monte-Carlo Method.
  • Markov Chain.
  • Planned Comparison.
• See: Data Analysis, Inductive Reasoning, Multiple Comparisons Problem, Clinical Research, Medical Statistics, ANOVA, Statistical Test, Subgroup Analysis, Exploratory Research, Statistical Population, Multivariate Hypothesis Testing.

---

---

References

2016

• (Wikipedia, 2016) ⇒ http://en.wikipedia.org/wiki/Post_hoc_analysis Retrieved 2016-08-28
  • In the design and analysis of experiments, post hoc analysis (from Latin post hoc, "after this") consists of looking at the data— after the experiment has concluded — for patterns that were not specified a priori. It is sometimes called data dredging by critics to evoke the sense that the more one looks the more likely something will be found. More subtly, each time a pattern in the data is considered, a statistical test is effectively performed. This greatly inflates the total number of statistical tests and necessitates the use of multiple testing procedures to compensate. However, this is difficult to do precisely and in fact most results of post hoc analyses are reported as they are with unadjusted p-values. These p-values must be interpreted in light of the fact that they are a small and selected subset of a potentially large group of p-values. Results of post hoc analyses should be explicitly labeled as such in reports and publications to avoid misleading readers.

In practice, post hoc analyses are usually concerned with finding patterns and/or relationships between subgroups of sampled populations that would otherwise remain undetected and undiscovered were a scientific community to rely strictly upon a priori statistical methods. Post hoc tests — also known as a posteriori tests — greatly expand the range and capability of methods that can be applied in exploratory research. Post hoc examination strengthens induction by limiting the probability that significant effects will seem to have been discovered between subgroups of a population when none actually exist. As it is, many scientific papers are published without adequate, preventative post hoc control of the type I error rate[1].

Post hoc analysis is an important procedure without which multivariate hypothesis testing would greatly suffer, rendering the chances of discovering false positives unacceptably high. Ultimately, post hoc testing creates better informed scientists who can therefore formulate better, more efficient a priori hypotheses and research designs.

1985

• (Kennedy & Bush, 1985) ⇒ Kennedy, J. J., & Bush, A. J. (1985). An introduction to the design and analysis of experiments in behavioral research. University Press of America. ISBN: 0-8191-4806-7
  • Unfortunately, most hypotheses in behavioral science research fail to meet the restrictions associated with a priori hypotheses. Consequently, we tend to make many comparisons in the anticipation that something interesting might materialize. The prevalent mode of operation is, therefore, to subject data to an overall analysis of variance, and should the F-test prove to be significant, to select appropriate post hoc procedures to determine the specific nature of treatment group differences. Since the choice of an appropriate post hoc test is important and is a problem which plagues many practitioners, post hoc procedures will be given considerable attention in this chapter (in: Chapter V, page 163).

---