Standardized effect sizes are far from “Standardized”: A primer and empirical illustration in depression psychotherapy meta-analyses

Standardized mean differences (SMDs) are frequently used to appraise the effects of psychological treatments, and to combine them in meta-analyses. Yet, there is no consensus on how exactly SMDs should be computed from randomized trials. In this study, we show that different SMD variants can heavily diverge in aggregate-data meta-analyses, subverting the original purpose of standardization. We investigate the impact this has on the estimated benefits of depression psychotherapies. Different SMD versions using endpoint or change scores were calculated from a comprehensive database of randomized trials, comparing depression psychotherapy against pharmacotherapy and inactive controls. Pooled treatment effects were obtained for each variant, assuming correlations between baseline and endpoint scores of 0.2 through 0.8, and their relationship was examined using bivariate meta-analyses. We also investigated which study characteristics predicted divergent effect estimates. A total of k = 443 trials with 48,221 participants were analyzed. The pooled effect of psychotherapy versus controls varied heavily depending on the calculation methods (SMD = 0.65–1.24), even though the same studies were used. Divergences were less pronounced for psychotherapies compared to pharmacotherapy (SMD = 0.05–0.14). Change score SMDs deviated from endpoint SMDs especially when high (r = 0.8) or low (r = 0.2) pre-post correlations were assumed. This difference was largest in subfields with high treatment effects. Different SMD calculation methods can lead to strongly diverging effect estimates of psychological treatment; especially when change scores are used and pre-post correlations are very high or low. This could have a profound impact on how treatment benefits are interpreted within and across meta-analyses. Researchers could prioritize endpoint SMDs of randomized trials, and should consider standardization using population-level estimates to improve the comparability of meta-analytic effects in the field.Open Material; Registration: htpps://doi.org/10.5281/zenodo.10694719; https://osf.io/yx5jg; https://osf.io/4j23t.