A Performance Analysis of Some New Meta-Analysis Estimators Designed to Correct Publication Bias

Publication selection bias is widely recognized as a serious challenge to the validity of meta-analyses. This study analyses the performance of three new estimators designed to correct publication bias: the weighted average of the adequately powered (WAAP) estimator of Stanley et al. (2017), and two estimators proposed by Andrews & Kasy (2019), which we call AK1 and AK2. With respect to bias, we find that none of these is consistently superior to the commonly used PET-PEESE estimator. With respect to mean squared error, we find that Andrews & Kasey’s AK1 estimator does consistently better than other estimators except when publication bias is focused solely on the sign, as opposed to the significance, of an effect. With respect to coverage rates, we find that all the estimators perform consistently poorly, so that hypothesis tests about the mean true effect are unreliable. We also find that effect heterogeneity generally worsens estimator performance, and that its adverse impact compounds with greater heterogeneity. This is particularly of concern for meta-analyses in business and economics, where I2 values, a measure of heterogeneity, are often 90 percent or higher. Finally, we find that the type of simulation environment used in the Monte Carlo experiments significantly impacts estimator performance. A better understanding of what makes an “appropriate” simulation environment for analysing meta-analysis estimators would be a potentially productive subject for future research.