Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023)

This study investigates the reproducibility and robustness of Stanley et al. (2023). Stanley et al. (2023) studied 67,308 meta-analyses of medical research from the Cochrane Database of Systematic Reviews (CDSR). They compared estimators using two information criteria: the Bayes Information Criterion (BIC) and the Akaike Information Criterion (AIC). They concluded that a variant of the Fixed Effect (FE) estimator they call Unrestricted Weighted Least Squares (UWLS) is “a better model of medical research than Random Effects (RE) regardless of heterogeneity, number of studies, or the type of outcome.” With respect to reproducibility, we can report that we are able to exactly reproduce Stanley et al. (2023)’s results. To test the robustness of their findings, we simulated 108,000 meta-analyses designed to represent the datasets in Stanley et al. (2023). This allowed us to assess estimator performance using bias, mean-squared error, and coverage rates. The overall conclusion we draw from our analysis is that BIC and AIC do not reliably guide the researcher to the “best” estimator when estimating mean treatment effects for medical research in Stanley et al. (2023)’s dataset. Further, the dominance of UWLS as measured by BIC and AIC is not matched by similar dominance on bias, MSE, and coverage rates. Most importantly, there is no evidence to support the headline result from Stanley et al. (2023) that UWLS dominates RE.