IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0298526.html
   My bibliography  Save this article

Do statistical heterogeneity methods impact the results of meta- analyses? A meta epidemiological study

Author

Listed:
  • Samer Mheissen
  • Haris Khan
  • David Normando
  • Nikhillesh Vaiid
  • Carlos Flores-Mir

Abstract

Background: Orthodontic systematic reviews (SRs) use different methods to pool the individual studies in a meta-analysis when indicated. However, the number of studies included in orthodontic meta-analyses is relatively small. This study aimed to evaluate the direction of estimate changes of orthodontic meta-analyses (MAs) using different between-study variance methods considering the level of heterogeneity when few trials were pooled. Methods: Search and study selection: Systematic reviews (SRs) published over the last three years, from the 1st of January 2020 to the 31st of December 2022, in six main orthodontic journals with at least one MA pooling five or lesser primary studies were identified. Data collection and analysis: Data were extracted from each eligible MA, which was replicated in a random effect model using DerSimonian and Laird (DL), Paule–Mandel (PM), Restricted maximum-likelihood (REML), Hartung Knapp and Sidik Jonkman (HKSJ) methods. The results were reported using median and interquartile range (IQR) for continuous data and frequencies for categorical data and analyzed using non-parametric tests. The Boruta algorithm was used to assess the significant predictors for the significant change in the confidence interval between the different methods compared to the DL method, which was only feasible using the HKSJ method. Results: 146 MAs were included, most applying the random effect model (n = 111; 76%) and pooling continuous data using mean difference (n = 121; 83%). The median number of studies was three (range 2, 4), and the overall statistical heterogeneity (I2 ranged from 0 to 99% with a median of 68%). Close to 60% of the significant findings became non-significant when HKSJ was applied compared to the DL method and when the heterogeneity was present I2>0%. On the other hand, 30.43% of the non-significant meta-analyses using the DL method became significant when HKSJ was used when the heterogeneity was absent I2 = 0%. Conclusion: Orthodontic MAs with few studies can produce different results based on the between-study variance method and the statistical heterogeneity level. Compared to DL, HKSJ method is overconservative when I2 is greater than 0% and may result in false positive findings when the heterogeneity is absent.

Suggested Citation

  • Samer Mheissen & Haris Khan & David Normando & Nikhillesh Vaiid & Carlos Flores-Mir, 2024. "Do statistical heterogeneity methods impact the results of meta- analyses? A meta epidemiological study," PLOS ONE, Public Library of Science, vol. 19(3), pages 1-14, March.
    • Handle: RePEc:plo:pone00:0298526
    DOI: 10.1371/journal.pone.0298526
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0298526
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0298526&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0298526?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sidik, Kurex & Jonkman, Jeffrey N., 2006. "Robust variance estimation for random effects meta-analysis," Computational Statistics & Data Analysis, Elsevier, vol. 50(12), pages 3681-3701, August.
    2. Kurex Sidik & Jeffrey N. Jonkman, 2005. "Simple heterogeneity variance estimation for meta‐analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 54(2), pages 367-384, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yu, Dalei & Ding, Chang & He, Na & Wang, Ruiwu & Zhou, Xiaohua & Shi, Lei, 2019. "Robust estimation and confidence interval in meta-regression models," Computational Statistics & Data Analysis, Elsevier, vol. 129(C), pages 93-118.
    2. Mathur, Maya B & VanderWeele, Tyler, 2017. "Sensitivity analysis for unmeasured confounding in meta-analyses," OSF Preprints jkhfg, Center for Open Science.
    3. Weber, Frank & Knapp, Guido & Ickstadt, Katja & Kundt, Günther & Glass, Anne, 2020. "Zero-cell corrections in random-effects meta-analyses," OSF Preprints qjh5f, Center for Open Science.
    4. repec:osf:osfxxx:jkhfg_v1 is not listed on IDEAS
    5. Johannes Hönekopp & Audrey Helen Linden, 2022. "Heterogeneity estimates in a biased world," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-21, February.
    6. Sidik, Kurex & Jonkman, Jeffrey N., 2006. "Robust variance estimation for random effects meta-analysis," Computational Statistics & Data Analysis, Elsevier, vol. 50(12), pages 3681-3701, August.
    7. repec:osf:metaar:3bnxs_v1 is not listed on IDEAS
    8. Amin Haghnejad & Mahboobeh Farahati, 2024. "An overview of meta-analytic methods for economic research," Papers 2412.10608, arXiv.org, revised Mar 2025.
    9. Lanju Zhang & Zailong Wang & Li Wang & Lu Cui & Jeremy Sokolove & Ivan Chan, 2022. "A Simple Approach to Incorporating Historical Control Data in Clinical Trial Design and Analysis," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 14(2), pages 216-236, July.
    10. Elena Kulinskaya & Stephan Morgenthaler & Robert G. Staudte, 2014. "Combining Statistical Evidence," International Statistical Review, International Statistical Institute, vol. 82(2), pages 214-242, August.
    11. repec:osf:thesis:kd6ja_v1 is not listed on IDEAS
    12. Weber, Frank & Knapp, Guido & Glass, Anne & Kundt, Günther & Ickstadt, Katja, 2020. "Interval estimation of the overall treatment effect in random-effects meta-analyses: Recommendations from a simulation study comparing frequentist, Bayesian, and bootstrap methods," OSF Preprints 5zbh6, Center for Open Science.
    13. Cojbasic, Vesna & Tomovic, Andrija, 2007. "Nonparametric confidence intervals for population variance of one sample and the difference of variances of two samples," Computational Statistics & Data Analysis, Elsevier, vol. 51(12), pages 5562-5578, August.
    14. Fengxia Dong & Wendy Zeng, 2024. "Effects of Fall and Winter Cover Crops on Weed Suppression in the United States: A Meta-Analysis," Sustainability, MDPI, vol. 16(8), pages 1-16, April.
    15. Bakbergenuly, Ilyas & Hoaglin, David C. & Kulinskaya, Elena, 2020. "Simulation study of estimating between-study variance and overall effect in meta-analyses of log-response-ratio for normal data," MetaArXiv 3bnxs, Center for Open Science.
    16. Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.
    17. Yannis Yan Liang & Mingqing Zhou & Yu He & Weijie Zhang & Qiqi Wu & Tong Luo & Jun Zhang & Fujun Jia & Lu Qi & Sizhi Ai & Jihui Zhang, 2024. "Observational and genetic evidence disagree on the association between loneliness and risk of multiple diseases," Nature Human Behaviour, Nature, vol. 8(11), pages 2209-2221, November.
    18. Yolanda Álvarez-Pérez & Amado Rivero-Santana & Lilisbeth Perestelo-Pérez & Andrea Duarte-Díaz & Vanesa Ramos-García & Ana Toledo-Chávarri & Alezandra Torres-Castaño & Beatriz León-Salas & Diego Infant, 2022. "Effectiveness of Mantra-Based Meditation on Mental Health: A Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 19(6), pages 1-18, March.
    19. Zhao, Changxun & Mo, Jiefei & Qin, Yong & Gong, Yabin & Zhan, Ouru & Song, Bo & Wu, Weixiang, 2025. "Semi-thermophilic anaerobic digestion (41–49°C): A review of its potential for enhanced methane production and system stability," Renewable and Sustainable Energy Reviews, Elsevier, vol. 215(C).
    20. Heinz Holling & Walailuck Böhning & Dankmar Böhning, 2012. "Likelihood-Based Clustering of Meta-Analytic SROC Curves," Psychometrika, Springer;The Psychometric Society, vol. 77(1), pages 106-126, January.
    21. Nick Drydakis, 2022. "Sexual orientation and earnings: a meta-analysis 2012–2020," Journal of Population Economics, Springer;European Society for Population Economics, vol. 35(2), pages 409-440, April.
    22. Zhichao Xu & Peng Wei, 2024. "A novel statistical framework for meta-analysis of total mediation effect with high-dimensional omics mediators in large-scale genomic consortia," PLOS Genetics, Public Library of Science, vol. 20(11), pages 1-23, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0298526. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.