P-Value Hacking Demo App

stat r shiny

An interactive demo app illustrating the NNT’s paper on P-value hacking.

In his paper (Taleb 2016), N.N.Taleb explains the idea of p-value hacking and derives the exact probability distribution for p-values across ensembles of statistically identical phenomena.

Some people whom I discussed that paper with, found it somewhat difficult to develop an intuition towards the practical consequences of p-values being a random variable with its own probability distribution.

To help develop such an intuition I’ve put together a very simple demo app using Shiny (Chang et al. 2022).

How to use the app:

  1. Read the original paper (Taleb 2016), this is a pre-requisite to using the app
  2. Open the app
  3. Try to move the sliders and observe the changes in the density plot, especially the shape of the distribution and the distance between median and expected p-values
  4. Contemplate the results.

The app’s source code is available at https://github.com/mikeivanov/pvalhack/.

Chang, Winston, Joe Cheng, JJ Allaire, Carson Sievert, Barret Schloerke, Yihui Xie, Jeff Allen, Jonathan McPherson, Alan Dipert, and Barbara Borges. 2022. Shiny: Web Application Framework for r. https://CRAN.R-project.org/package=shiny.
Taleb, Nassim Nicholas. 2016. “A Short Note on p-Value Hacking.” arXiv. https://doi.org/10.48550/ARXIV.1603.07532.

References

© Copyright 2021 Mike Ivanov