Forking paths problem
The garden of forking paths is a problem in frequentist hypothesis testing through which researchers can unintentionally produce false positives for a tested hypothesis, through leaving themselves too many degrees of freedom. In contrast to fishing expeditions such as data dredging where only expected or apparently-significant results are published, this allows for a similar effect even when only one experiment is run, through a series of choices about how to implement methods and analyses, which are themselves informed by the data as it is observed and processed.[1]
History
Exploring a forking decision-tree while analyzing data was at one point grouped with the multiple comparisons problem as an example of poor statistical method. However Gelman and Loken demonstrated[2] that this can happen implicitly by researchers aware of best practices who only make a single comparison and only evaluate their data once.
The fallacy is believing an analysis to be free of multiple comparisons despite having had enough degrees of freedom in choosing the method, after seeing some or all of the data, to produce similarly-grounded false positives. Degrees of freedom can include choosing among main effects or interactions, methods for data exclusion, whether to combine different studies, and method of data analysis.
See also
References
- ↑ "Garden of forking paths" (in en-us). https://forrt.org/glossary/garden-of-forking-paths/.
- ↑ Gelman, Andrew; Loken, Eric (November 14, 2013). "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time". http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.
 |  |
