Leakage explains the apparent superiority of Bayesian random effect models – a preregistered comment on Claessens, Kyritsis and Atkinson (2023)

In a previous study, Claessens, Kyritsis, and Atkinson (CKA) demonstrated the importance of controlling for geographic proximity and cultural similarity in cross-national analyses. Based on a simulation study, CKA showed that methods commonly used to control for spatial and cultural non-independence are insufficient in reducing false positives while maintaining the ability to detect true effects. CKA strongly advocate the use of Bayesian random effect models in such situations, arguing that among the studied model types, they are the only ones that reduced false positives while maintaining high statistical power. However, in this comment, we argue that the apparent superiority of such models is overstated by CKA due to a form of methodological circularity called 'leakage' in statistics and machine learning, because the same proximity matrix is used both to generate the simulated data and as an input to only the Bayesian models for comparison. When this leakage is controlled for, we show that Bayesian models do not outperform most other methods.