When is p-hacking detectable?

Some forms of p-hacking cannot be detected by examining the t-curve (or p-curve). Standard tests may also fail to find even detectable forms of selective reporting. We propose a novel test that is consistent against every detectable form of p-hacking and remains interpretable even when the t-scores are not exactly normal. The test statistic is the distance between the smoothed empirical t-curve and the set of all distributions that would be possible in the absence of any selective reporting. This novel projection test can only be evaded in large meta-samples by selective reporting that also evades all other valid tests of restrictions on the t-curve. A second benefit of the projection test is that under the null hypothesis of no p-hacking we can check whether the projection residual could have been produced by other distortions not related to selective reporting, e.g. rounding and de-rounding. Applying the test to the Brodeur et al. (2020) meta-data, we find that the t-curves for RCTs, IVs, and DIDs are more distorted than could arise by chance. We confirm that these distortions cannot be explained by (de)rounding of t-scores or by the limited degrees of freedom of the underlying studies.