IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v40y2015i6p604-634.html

Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression

Author

Listed:
  • Elizabeth Tipton

    (Columbia University)

  • James E. Pustejovsky

    (University of Texas at Austin)

Abstract

Meta-analyses often include studies that report multiple effect sizes based on a common pool of subjects or that report effect sizes from several samples that were treated with very similar research protocols. The inclusion of such studies introduces dependence among the effect size estimates. When the number of studies is large, robust variance estimation (RVE) provides a method for pooling dependent effects, even when information on the exact dependence structure is not available. When the number of studies is small or moderate, however, test statistics and confidence intervals based on RVE can have inflated Type I error. This article describes and investigates several small-sample adjustments to F -statistics based on RVE. Simulation results demonstrate that one such test, which approximates the test statistic using Hotelling’s T 2 distribution, is level-α and uniformly more powerful than the others. An empirical application demonstrates how results based on this test compare to the large-sample F -test.

Suggested Citation

  • Elizabeth Tipton & James E. Pustejovsky, 2015. "Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression," Journal of Educational and Behavioral Statistics, , vol. 40(6), pages 604-634, December.
    • Handle: RePEc:sae:jedbes:v:40:y:2015:i:6:p:604-634
    DOI: 10.3102/1076998615606099
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998615606099
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998615606099?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Xiaohong Chen & Norman R. Swanson (ed.), 2013. "Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis," Springer Books, Springer, edition 127, number 978-1-4614-1653-1, January.
    2. Eric C. Hedberg, 2011. "ROBUMETA: Stata module to perform robust variance estimation in meta-regression with dependent effect size estimates," Statistical Software Components S457219, Boston College Department of Economics, revised 23 Apr 2014.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2023. "Fast and reliable jackknife and bootstrap methods for cluster‐robust inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 671-694, August.
    2. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LLC, vol. 19(1), pages 4-60, March.
    3. Molinari, Francesca, 2020. "Microeconometrics with partial identification," Handbook of Econometrics, in: Steven N. Durlauf & Lars Peter Hansen & James J. Heckman & Rosa L. Matzkin (ed.), Handbook of Econometrics, edition 1, volume 7, chapter 0, pages 355-486, Elsevier.
    4. Demian Pouzo, 2015. "On the Non-Asymptotic Properties of Regularized M-estimators," Papers 1512.06290, arXiv.org, revised Oct 2016.
    5. Tomaz Cajner & Leland D. Crane & Ryan A. Decker & Adrian Hamins-Puertolas & Christopher Kurz, 2019. "Improving the Accuracy of Economic Measurement with Multiple Data Sources: The Case of Payroll Employment Data," NBER Chapters, in: Big Data for Twenty-First-Century Economic Statistics, pages 147-170, National Bureau of Economic Research, Inc.
    6. Mehmet Pinar & Thanasis Stengos & Nikolas Topaloglou, 2022. "Stochastic dominance spanning and augmenting the human development index with institutional quality," Annals of Operations Research, Springer, vol. 315(1), pages 341-369, August.
    7. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    8. P. Dorian Owen, 2017. "Evaluating Ingenious Instruments for Fundamental Determinants of Long-Run Economic Growth and Development," Econometrics, MDPI, vol. 5(3), pages 1-33, September.
    9. Daouia, Abdelaati & Laurent, Thibault & Noh, Hohsuk, 2017. "npbr: A Package for Nonparametric Boundary Regression in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 79(i09).
    10. Sylvain Barde, 2015. "A fast algorithm for finding the confidence set of large collections of models," Studies in Economics 1519, School of Economics, University of Kent.
    11. Francis X. Diebold & Minchul Shin, 2017. "Assessing point forecast accuracy by stochastic error distance," Econometric Reviews, Taylor & Francis Journals, vol. 36(6-9), pages 588-598, October.
    12. Richard, Patrick, 2019. "Residual bootstrap tests in linear models with many regressors," Journal of Econometrics, Elsevier, vol. 208(2), pages 367-394.
    13. Dan Gabriel Anghel, 2015. "Market Efficiency and Technical Analysis in Romania," International Journal of Financial Research, International Journal of Financial Research, Sciedu Press, vol. 6(2), pages 164-177, April.
    14. Pustejovsky, James E & Tipton, Elizabeth, 2020. "Meta-Analysis with Robust Variance Estimation: Expanding the Range of Working Models," MetaArXiv vyfcj, Center for Open Science.
    15. Norman R. Swanson & Weiqi Xiong, 2018. "Big data analytics in economics: What have we learned so far, and where should we go from here?," Canadian Journal of Economics, Canadian Economics Association, vol. 51(3), pages 695-746, August.
    16. Valentina Corradi & Sainan Jin & Norman R. Swanson, 2023. "Robust forecast superiority testing with an application to assessing pools of expert forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(4), pages 596-622, June.
    17. Solat, Karo & Tsang, Kwok Ping, 2021. "Forecasting exchange rates with elliptically symmetric principal components," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1085-1091.
    18. Steven Windisch & Susann Wiedlitzka & Ajima Olaghere, 2021. "PROTOCOL: Online interventions for reducing hate speech and cyberhate: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(1), March.
    19. Pötscher, Benedikt M. & Preinerstorfer, David, 2025. "Valid Heteroskedasticity Robust Testing," Econometric Theory, Cambridge University Press, vol. 41(2), pages 249-301, April.
    20. Erica K. Chuang & Barbara S. Mensch & Stephanie R. Psaki & Nicole A. Haberland & Meredith L. Kozak, 2019. "PROTOCOL: Policies and interventions to remove gender‐related barriers to girls’ school participation and learning in low‐ and middle‐income countries: A systematic review of the evidence," Campbell Systematic Reviews, John Wiley & Sons, vol. 15(3), September.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:40:y:2015:i:6:p:604-634. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.