Sensitivity analysis for p-hacking in meta-analyses

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., significant, positive estimates) versus nonaffirmative studies (i.e., nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators “p-hack’’ results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, compromising existing methods that only consider SAS. We propose two sensitivity analyses that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, assuming that published, hacked studies never have nonaffirmative estimates (e.g., their investigators p-hack until they obtain an affirmative estimate), we propose estimating the underlying meta-analytic mean by fitting “right-truncated meta-analysis’’ (RTMA) to the published nonaffirmative estimates, which are unhacked. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions, including when nonaffirmative estimates from p-hacked studies are sometimes published. We provide an R package, phacking. Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.