Many ask ‘what’s wrong with p-value’ nowadays after the replicability/reproducibility crisis particularly was noticed in the field of psychology. It sparked a discussion within the statistical community and eventually reached the point where the American Statistical Association (ASA) had to make a comment on the practice of using the p-value (this is a rare case… they don’t usually do this)
So what’s so wrong with the p-value? Why is it pinpointed to be one of the causes of the crisis? I want to give an intuitive answer to it.
Because it’s gonna give you the wrong result percent of the time.
Whoa whoa calm down…. This post wasn’t contrived to trigger an online war with the frequentists. But really, we all need to stop taking it for granted and try to appreciate what that really means. What would really happen if it gave you the wrong result that many times?
Ok so suppose there is an academic group consisting of thousands of researchers striving to publish their papers and they need to present an eye-opening alternative hypothesis being accepted over the null hypothesis (in other words, the result is statistically significant). AND! think about a situation where every single one of them imagines a complete bullshit and puts it as their alternative. EVERY SINGLE ONE!
This kinda seems what’s happening. Really. If they set to be their significance level, 5% of those bullshits will be accepted by chance.
Adding insult to injury (LOL old saying), here comes the publication bias. What do you think is going to get published: the boring & all-to-obvious true results whose tests have failed to reject the true null vs. the shocking news that happens to accidentally have passed the test? OF COURSE THE LATTER! Now this is the real problem. Those researchers who couldn’t publish their papers will never inform the public of their true results and the public will unwittingly take bullshit as their truth.
I know this imaginary situation may seem unlikely but it is not as unlikely as you might think. I hear these annoyingly false results from the field of medicine and they’re pretty much every time from the U.K. and I don’t know why. The news is really shocking because the hypotheses that get tested are shocking in the first place and I personally believe them to be shenanigans / canards, the very examples of bullshit passing the test. (For example, men with big feet have big penises…. Do you seriously believe that?)
I really don’t have a say in the discussion of “what to do next” because I’m not a fully-educated statistician. But I would recommend reporting more information than just throwing p-values here and there. And most importantly, always keep in mind that your data & hypotheses could be subject to accidental falsehood. It’s time to stop this p-value nonsense.