IDEAS home Printed from https://ideas.repec.org/a/bla/jorssc/v69y2020i5p1091-1119.html
   My bibliography  Save this article

Sensitivity analysis for publication bias in meta‐analyses

Author

Listed:
  • Maya B. Mathur
  • Tyler J. VanderWeele

Abstract

We propose sensitivity analyses for publication bias in meta‐analyses. We consider a publication process such that ‘statistically significant’ results are more likely to be published than negative or “non‐significant” results by an unknown ratio, η. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse probability weighting and robust estimation that accommodates non‐normal population effects, small meta‐analyses, and clustering, we develop sensitivity analyses that enable statements such as ‘For publication bias to shift the observed point estimate to the null, “significant” results would need to be at least 30 fold more likely to be published than negative or “non‐significant” results’. Comparable statements can be made regarding shifting to a chosen non‐null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of η across disciplines. We show that a worst‐case meta‐analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta‐analysis of only the negative and ‘non‐significant’ studies; this method sometimes indicates that no amount of such publication bias could ‘explain away’ the results. We illustrate the proposed methods by using real meta‐analyses and provide an R package: PublicationBias.

Suggested Citation

  • Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
  • Handle: RePEc:bla:jorssc:v:69:y:2020:i:5:p:1091-1119
    DOI: 10.1111/rssc.12440
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssc.12440
    Download Restriction: no

    File URL: https://libkey.io/10.1111/rssc.12440?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Blakeley B. McShane & David Gal, 2017. "Rejoinder: Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 904-908, July.
    2. Valen E. Johnson & Richard D. Payne & Tianying Wang & Alex Asher & Soutrik Mandal, 2017. "On the Reproducibility of Psychological Science," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(517), pages 1-10, January.
    3. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    4. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    5. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    6. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    7. Blakeley B. McShane & David Gal, 2017. "Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 885-895, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Irsova, Zuzana & Bom, Pedro R. D. & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," EconStor Preprints 268683, ZBW - Leibniz Information Centre for Economics.
    2. Nina T. Dalgaard & Anja Bondebjerg & Bjørn C. A. Viinholt & Trine Filges, 2022. "The effects of inclusion on academic achievement, socioemotional development and wellbeing of children with special educational needs," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(4), December.
    3. Jens Dietrichson & Morten Kjær Thomsen & Julie Kaas Seerup & Martin Williams Strandby & Bjørn Christian Arleth Viinholt & Elizabeth Bengtsen, 2022. "PROTOCOL: School‐based language, math, and reading interventions for executive functions in children and adolescents: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    4. Maier, Maximilian & VanderWeele, Tyler & Mathur, Maya B, 2021. "Using Selection Models to Assess Sensitivity to Publication Bias: A Tutorial and Call for More Routine Use," MetaArXiv tp45u, Center for Open Science.
    5. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    6. Ao Huang & Kosuke Morikawa & Tim Friede & Satoshi Hattori, 2023. "Adjusting for publication bias in meta‐analysis via inverse probability weighting using clinical trial registries," Biometrics, The International Biometric Society, vol. 79(3), pages 2089-2102, September.
    7. Trine Filges & Geir Smedslund & Tine Eriksen & Kirsten Birkefoss, 2023. "PROTOCOL: The FRIENDS preventive programme for reducing anxiety symptoms in children and adolescents: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 19(4), December.
    8. Anja Bondebjerg & Nina T. Dalgaard & Trine Filges & Morten K. Thomsen & Bjørn C. A. Viinholt, 2021. "PROTOCOL: The effects of small class sizes on students’ academic achievement, socioemotional development, and well‐being in special education," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(2), June.
    9. Anja Bondebjerg & Nina Thorup Dalgaard & Trine Filges & Bjørn Christian Arleth Viinholt, 2023. "The effects of small class sizes on students' academic achievement, socioemotional development and well‐being in special education: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 19(3), September.
    10. Nina T. Dalgaard & Anja Bondebjerg & Bjørn C. A. Viinholt & Trine Filges, 2021. "PROTOCOL: The effects of inclusion on academic achievement, socioemotional development and wellbeing of children with special educational needs," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(2), June.
    11. Maximilian Maier & Tyler J. VanderWeele & Maya B. Mathur, 2022. "Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    12. Morten K. Thomsen & Julie K. Seerup & Jens Dietrichson & Anja Bondebjerg & Bjørn C. A. Viinholt, 2022. "PROTOCOL: Testing frequency and student achievement: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(1), March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    2. Maximilian Maier & Tyler J. VanderWeele & Maya B. Mathur, 2022. "Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    3. Sanghyun Hong & W. Robert Reed, 2020. "Using Monte Carlo Experiments to Select Meta-Analytic Estimators," Working Papers in Economics 20/10, University of Canterbury, Department of Economics and Finance.
    4. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    5. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    6. Jonas Moss & Riccardo De Bin, 2023. "Modelling publication bias and p‐hacking," Biometrics, The International Biometric Society, vol. 79(1), pages 319-331, March.
    7. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    8. Higney, Anthony & Hanley, Nick & Moro, Mirko, 2022. "The lead-crime hypothesis: A meta-analysis," Regional Science and Urban Economics, Elsevier, vol. 97(C).
    9. Bertoldi, Paolo & Mosconi, Rocco, 2020. "Do energy efficiency policies save energy? A new approach based on energy policy indicators (in the EU Member States)," Energy Policy, Elsevier, vol. 139(C).
    10. Maier, Maximilian & VanderWeele, Tyler & Mathur, Maya B, 2021. "Using Selection Models to Assess Sensitivity to Publication Bias: A Tutorial and Call for More Routine Use," MetaArXiv tp45u, Center for Open Science.
    11. Anderson, Brian S. & Wennberg, Karl & McMullen, Jeffery S., 2019. "Editorial: Enhancing quantitative theory-testing entrepreneurship research," Journal of Business Venturing, Elsevier, vol. 34(5), pages 1-1.
    12. Wennberg, Karl & Anderson, Brian S. & McMullen, Jeffrey, 2019. "2 Editorial: Enhancing Quantitative Theory-Testing Entrepreneurship Research," Ratio Working Papers 323, The Ratio Institute.
    13. David J. Hand, 2022. "Trustworthiness of statistical inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(1), pages 329-347, January.
    14. Anderson, Brian S., 2022. "What executives get wrong about statistics: Moving from statistical significance to effect sizes and practical impact," Business Horizons, Elsevier, vol. 65(3), pages 379-388.
    15. Hong, Sanghyun & Robert Reed, W. & Tian, Bifei & Wu, Tingting & Chen, Gen, 2021. "Does FDI promote entrepreneurial activities? A meta-analysis," World Development, Elsevier, vol. 142(C).
    16. J. M. Bauer & L. A. Reisch, 2019. "Behavioural Insights and (Un)healthy Dietary Choices: a Review of Current Evidence," Journal of Consumer Policy, Springer, vol. 42(1), pages 3-45, March.
    17. Jeffrey A. Mills & Gary Cornwall & Beau A. Sauley & Jeffrey R. Strawn, 2018. "Improving the Analysis of Randomized Controlled Trials: a Posterior Simulation Approach," BEA Working Papers 0157, Bureau of Economic Analysis.
    18. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    19. Han Wang & Sieglinde S Snapp & Monica Fisher & Frederi Viens, 2019. "A Bayesian analysis of longitudinal farm surveys in Central Malawi reveals yield determinants and site-specific management strategies," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-17, August.
    20. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssc:v:69:y:2020:i:5:p:1091-1119. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.