IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v40y2015i6p604-634.html
   My bibliography  Save this article

Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression

Author

Listed:
  • Elizabeth Tipton

    (Columbia University)

  • James E. Pustejovsky

    (University of Texas at Austin)

Abstract

Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.

Suggested Citation

  • Elizabeth Tipton & James E. Pustejovsky, 2015. "Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression," Journal of Educational and Behavioral Statistics, , vol. 40(6), pages 604-634, December.
    • Handle: RePEc:sae:jedbes:v:40:y:2015:i:6:p:604-634
    DOI: 10.3102/1076998615606099
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998615606099
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998615606099?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Xiaohong Chen & Norman R. Swanson (ed.), 2013. "Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis," Springer Books, Springer, edition 127, number 978-1-4614-1653-1, November.
    2. Eric C. Hedberg, 2011. "ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates," Statistical Software Components S457219, Boston College Department of Economics, revised 23 Apr 2014.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nina T. Dalgaard & Maya C. Flensborg Jensen & Elizabeth Bengtsen & Karl F. Krassel & Mikkel H. Vembye, 2022. "PROTOCOL: Group‐based community interventions to support the social reintegration of marginalised adults with mental illness," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    2. Jens Dietrichson & Morten Kjær Thomsen & Julie Kaas Seerup & Martin Williams Strandby & Bjørn Christian Arleth Viinholt & Elizabeth Bengtsen, 2022. "PROTOCOL: School‐based language, math, and reading interventions for executive functions in children and adolescents: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    3. Jared Coopersmith & Thomas D. Cook & Jelena Zurovac & Duncan Chaplin & Lauren V. Forrow, 2022. "Internal And External Validity Of The Comparative Interrupted Time‐Series Design: A Meta‐Analysis," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 41(1), pages 252-277, January.
    4. Welz, Thilo & Viechtbauer, Wolfgang & Pauly, Markus, 2023. "Cluster-robust estimators for multivariate mixed-effects meta-regression," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    5. Pustejovsky, James E & Tipton, Elizabeth, 2020. "Meta-Analysis with Robust Variance Estimation: Expanding the Range of Working Models," MetaArXiv vyfcj, Center for Open Science.
    6. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    7. Ciara Keenan & Sarah Miller & Jennifer Hanratty & Therese D. Pigott & Peter Mackie & John Cowman & Christopher Coughlan & Jayne Hamilton & Suzanne Fitzpatrick, 2020. "PROTOCOL: Accommodation‐based interventions for individuals experiencing, or at risk of experiencing, homelessness," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(3), September.
    8. Vembye, Mikkel Helding & Pustejovsky, James E & Pigott, Terri, 2022. "Power Approximations for Meta-Analysis of Dependent Effect Sizes," MetaArXiv 6tp9y, Center for Open Science.
    9. Tyler E. Smith & Aaron M. Thompson & Brandy R. Maynard, 2022. "Self‐management interventions for reducing challenging behaviors among school‐age students: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(1), March.
    10. Jens Dietrichson & Trine Filges & Rasmus H. Klokker & Bjørn C. A. Viinholt & Martin Bøg & Ulla H. Jensen, 2020. "Targeted school‐based interventions for improving reading and mathematics for students with, or at risk of, academic difficulties in Grades 7–12: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(2), June.
    11. Jennifer Hanratty & Ciara Keenan & Sean O'Connor & Sarah Miller & Declan Bradley & Martin Dempster, 2022. "PROTOCOL: Psychological and psychosocial determinants of COVID Health Related Behaviours (COHeRe): A suite of systematic reviews and an evidence and gap map," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(1), March.
    12. Sarah Miller & Ciara Keenan & Jennifer Hanratty & Jayne Hamilton & Christopher Coughlan & Peter Mackie & Suzanne Fitzpatrick & Alan Maddock, 2020. "PROTOCOL: Improving access to health and social services for individuals experiencing, or at risk of experiencing, homelessness," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(4), December.
    13. John Protzko & Jon Krosnick & Leif Nelson & Brian A. Nosek & Jordan Axt & Matt Berent & Nicholas Buttrick & Matthew DeBell & Charles R. Ebersole & Sebastian Lundmark & Bo MacInnis & Michael O’Donnell , 2024. "High replicability of newly discovered social-behavioural findings is achievable," Nature Human Behaviour, Nature, vol. 8(2), pages 311-319, February.
    14. Jennifer Hanratty & Sarah Miller & Ciara Keenan & John Cowman & Jayne Hamilton & Peter Mackie, 2020. "PROTOCOL: Discharge programmes for individuals experiencing, or at risk of experiencing, homelessness: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(3), September.
    15. Waheed, Hassam, 2023. "Nudging smokers away from lighting up: A meta-analysis of framing effect in current smokers," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 104(C).
    16. Ciara Keenan & Sarah Miller & Jennifer Hanratty & Terri Pigott & Jayne Hamilton & Christopher Coughlan & Peter Mackie & Suzanne Fitzpatrick & John Cowman, 2021. "Accommodation‐based interventions for individuals experiencing, or at risk of experiencing, homelessness," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(2), June.
    17. Latha Velayudhan & Katie McGoohan & Sagnik Bhattacharyya, 2021. "Safety and tolerability of natural and synthetic cannabinoids in adults aged over 50 years: A systematic review and meta-analysis," PLOS Medicine, Public Library of Science, vol. 18(3), pages 1-42, March.
    18. Mikkel Helding Vembye & James Eric Pustejovsky & Therese Deocampo Pigott, 2023. "Power Approximations for Overall Average Effects in Meta-Analysis With Dependent Effect Sizes," Journal of Educational and Behavioral Statistics, , vol. 48(1), pages 70-102, February.
    19. Morten K. Thomsen & Julie K. Seerup & Jens Dietrichson & Anja Bondebjerg & Bjørn C. A. Viinholt, 2022. "PROTOCOL: Testing frequency and student achievement: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(1), March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2023. "Fast and reliable jackknife and bootstrap methods for cluster‐robust inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 671-694, August.
    2. Norman R. Swanson & Weiqi Xiong, 2018. "Big data analytics in economics: What have we learned so far, and where should we go from here?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 51(3), pages 695-746, August.
    3. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LP, vol. 19(1), pages 4-60, March.
    4. Pötscher, Benedikt M. & Preinerstorfer, David, 2021. "Valid Heteroskedasticity Robust Testing," MPRA Paper 107420, University Library of Munich, Germany.
    5. Eiji Goto & Jan P.A.M. Jacobs & Tara M. Sinclair & Simon van Norden, 2023. "Employment reconciliation and nowcasting," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(7), pages 1007-1017, November.
    6. Francesca Molinari, 2020. "Microeconometrics with Partial Identi?cation," CeMMAP working papers CWP15/20, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    7. Demian Pouzo, 2015. "On the Non-Asymptotic Properties of Regularized M-estimators," Papers 1512.06290, arXiv.org, revised Oct 2016.
    8. Pötscher, Benedikt M. & Preinerstorfer, David, 2023. "How Reliable Are Bootstrap-Based Heteroskedasticity Robust Tests?," Econometric Theory, Cambridge University Press, vol. 39(4), pages 789-847, August.
    9. Francis X. Diebold & Minchul Shin, 2017. "Assessing point forecast accuracy by stochastic error distance," Econometric Reviews, Taylor & Francis Journals, vol. 36(6-9), pages 588-598, October.
    10. Tomaz Cajner & Leland D. Crane & Ryan A. Decker & Adrian Hamins-Puertolas & Christopher Kurz, 2019. "Improving the Accuracy of Economic Measurement with Multiple Data Sources: The Case of Payroll Employment Data," NBER Chapters, in: Big Data for Twenty-First-Century Economic Statistics, pages 147-170, National Bureau of Economic Research, Inc.
    11. Mehmet Pinar & Thanasis Stengos & Nikolas Topaloglou, 2022. "Stochastic dominance spanning and augmenting the human development index with institutional quality," Annals of Operations Research, Springer, vol. 315(1), pages 341-369, August.
    12. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    13. P. Dorian Owen, 2017. "Evaluating Ingenious Instruments for Fundamental Determinants of Long-Run Economic Growth and Development," Econometrics, MDPI, vol. 5(3), pages 1-33, September.
    14. Daouia, Abdelaati & Laurent, Thibault & Noh, Hohsuk, 2017. "npbr: A Package for Nonparametric Boundary Regression in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 79(i09).
    15. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    16. Sylvain Barde, 2015. "A fast algorithm for finding the confidence set of large collections of models," Studies in Economics 1519, School of Economics, University of Kent.
    17. Richard, Patrick, 2019. "Residual bootstrap tests in linear models with many regressors," Journal of Econometrics, Elsevier, vol. 208(2), pages 367-394.
    18. Dan Gabriel Anghel, 2015. "Market Efficiency and Technical Analysis in Romania," International Journal of Financial Research, International Journal of Financial Research, Sciedu Press, vol. 6(2), pages 164-177, April.
    19. Pustejovsky, James E & Tipton, Elizabeth, 2020. "Meta-Analysis with Robust Variance Estimation: Expanding the Range of Working Models," MetaArXiv vyfcj, Center for Open Science.
    20. Marijn Verschelde & Michel Dumont & Glenn Rayp & Bruno Merlevede, 2016. "Semiparametric stochastic metafrontier efficiency of European manufacturing firms," Journal of Productivity Analysis, Springer, vol. 45(1), pages 53-69, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:40:y:2015:i:6:p:604-634. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.