Towards an Experimental Framework for Assessing Meta-Analysis Methods, with a Focus on Andrews-Kasy Estimators

The study contributes towards the development of a systematic experimental framework for evaluating meta-analysis methods. Towards that goal, we reproduce the Monte Carlo experiments from three studies: Stanley & Doucouliagos (2017); Stanley, Doucouliagos, & Ioannidis (2017); and Alinaghi & Reed (2018) – S&D, SD&I, and A&R, respectively. We demonstrate that the relative performance of estimators depends on whether the researcher is concerned with unbiasedness, mean squared error (MSE), or coverage rates. We also show how estimator performance varies systematically with the number of estimates in the meta-analyst’s sample and the degree of effect heterogeneity as measured by I2. This demonstrates the possibility that researchers can select a “best estimator” based on the observable characteristics of their meta-analysis samples. We further show that the design of simulation experiments makes a difference: Different simulation designs by S&D and SD&I applied to the same “types” of meta-analysis samples select different “best” estimators. Different designs in A&R also produce different results. This highlights the need to know more about which aspects of simulation designs are important for estimator performance. Finally, our results indicate that the recent Andrews & Kasy (2019) estimators perform well in a number of research environments, frequently outperforming the popular PET-PEESE and WAAP estimators, though more research is needed.