Reproduction Report on "Unpacking p-Hacking and Publication Bias"

Brodeur, Carrell, Figlio and Lusher (2023) present the first descriptive evidence on p-hacking and publication bias throughout the publication process in economics. Using data from the Journal of Human Resources, they show that 1) p-values in initial submissions present humping around the 10 and 5% significance thresholds, 2) the distribution of desk rejections shows more humping than that of papers sent for review, 3) favorable reviewer recommendations are positively related to statistical significance and 4) overall, the p-value distribution of initial and final versions of accepted drafts appear similar. We first conduct a mostly successful computational reproduction of those results. Next, we conduct a series of robustness replication exercises and show that, while the authors' main results are robust to random removals of small amounts of data and to bandwidth choices, they are sensitive to removing papers that contribute many estimates, to some key modelling choices in density discontinuity tests, and to some conceptually equivalent variations of their tests. Last, we complement the authors' results by providing statistical tests of their main hypotheses. Our results support the claims that 1) the distribution of p-values of deskrejected papers displays greater bunching than that of papers sent for review, and 2) the distributions of p-values of initial submissions and final drafts are similar. Overall, our findings lend support to the authors' main claims, suggesting that after excluding desk-rejected papers, the peer review process does not seem to affect p-hacking or publication bias.