Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023)

This study critically examines the reproducibility and robustness of Stanley et al. (2023). Stanley et al. analyzed 67,308 meta-analysis datasets from the Cochrane Database of Systematic Reviews (CDSR). They find that a generalized Fixed Effect estimator they call unrestricted weighted least squares (UWLS-FE) is determined by the model selection criteria AIC and BIC to be superior to Random Effects (RE). In some cases, it is also better than Fixed Effects. Our analysis uses their original data and code and exactly reproduces their results. However, we show that the small sample sizes of the CDSR datasets undermine the reliability of AIC and BIC for model selection. Accordingly, we simulate 108,000 datasets mirroring the original CDSR data. This allows us to know the true model parameters and evaluate the estimators more accurately. Our findings suggest that RE generally outperforms UWLS-FE. The comparison with FE is less clear. FE frequently produces more accurate standard errors than UWLS-FE, making confidence intervals and hypothesis testing more reliable. However, for some types of data, UWLS-FE may be better than FE.