IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/20-10.html
   My bibliography  Save this paper

Using Monte Carlo Experiments to Select Meta-Analytic Estimators

Author

Abstract

The purpose of this study is to show how Monte Carlo analysis of meta-analytic estimators can be used to select estimators for specific research situations. Our analysis conducts 1,620 individual experiments, where each experiment is defined by a unique combination of sample size, effect heterogeneity, effect size, publication selection mechanism, and other research characteristics. We compare eleven estimators commonly used in medicine, psychology, and the social sciences. These are evaluated on the basis of bias, mean squared error (MSE), and coverage rates. For our experimental design, we reproduce simulation environments from four recent studies: Stanley, Doucouliagos, & Ioannidis (2017), Alinaghi & Reed (2018), Bom & Rachinger (2019), and Carter et al. (2019a). We demonstrate that relative estimator performance differs across performance measures. An estimator that may be especially good with respect to MSE may perform relatively poorly with respect to coverage rates. We also show that sample size and effect heterogeneity are important determinants of relative estimator performance. We use these results to demonstrate how the observable characteristics of sample size and effect heterogeneity can guide the meta-analyst in choosing the estimators most appropriate for their research circumstances. All of the programming code and output files associated with this project are available at https://osf.io/pr4mb/.

Suggested Citation

  • Sanghyun Hong & W. Robert Reed, 2020. "Using Monte Carlo Experiments to Select Meta-Analytic Estimators," Working Papers in Economics 20/10, University of Canterbury, Department of Economics and Finance.
  • Handle: RePEc:cbt:econwp:20/10
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/2010.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    2. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    3. Nazila Alinaghi & W. Robert Reed, 2016. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 16/26, University of Canterbury, Department of Economics and Finance.
    4. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    5. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    6. Roger M. Harbord & Ross J. Harris & Jonathan A. C. Sterne, 2009. "Updated tests for small-study effects in meta-analyses," Stata Journal, StataCorp LP, vol. 9(2), pages 197-210, June.
    7. Reed, W. Robert, 2015. "A Monte Carlo analysis of alternative meta-analysis estimators in the presence of publication bias," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-40.
    8. J. Copas, 1999. "What works?: selectivity models and meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(1), pages 95-109.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nazila Alinaghi & W. Robert Reed, 2021. "Taxes and Economic Growth in OECD Countries: A Meta-analysis," Public Finance Review, , vol. 49(1), pages 3-40, January.
    2. Hong, Sanghyun & Robert Reed, W. & Tian, Bifei & Wu, Tingting & Chen, Gen, 2021. "Does FDI promote entrepreneurial activities? A meta-analysis," World Development, Elsevier, vol. 142(C).
    3. Ivan Ropovik & Matus Adamkovic & David Greger, 2021. "Neglect of publication bias compromises meta-analyses of educational research," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-14, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    2. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    3. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    4. Sanghyun Hong & W. Robert Reed, 2019. "A Performance Analysis of Some New Meta-Analysis Estimators Designed to Correct Publication Bias," Working Papers in Economics 19/04, University of Canterbury, Department of Economics and Finance.
    5. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    6. Sanghyun Hong & W. Robert Reed, 2019. "Towards an Experimental Framework for Assessing Meta-Analysis Methods, with a Focus on Andrews-Kasy Estimators," Working Papers in Economics 19/13, University of Canterbury, Department of Economics and Finance.
    7. Nelson, Jon Paul, 2020. "Fixed-effect versus random-effects meta-analysis in economics: A study of pass-through rates for alcohol beverage excise taxes," Economics Discussion Papers 2020-1, Kiel Institute for the World Economy (IfW Kiel).
    8. Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
    9. Marvin Schütt, 2021. "Systematic Variation in Waste Site Effects on Residential Property Values: A Meta-Regression Analysis and Benefit Transfer," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 78(3), pages 381-416, March.
    10. Xue, Xindong & Reed, W. Robert & Menclova, Andrea, 2020. "Social capital and health: a meta-analysis," Journal of Health Economics, Elsevier, vol. 72(C).
    11. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    12. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    13. Nazila Alinaghi & W. Robert Reed, 2016. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 16/26, University of Canterbury, Department of Economics and Finance.
    14. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    15. Jonas Moss & Riccardo De Bin, 2023. "Modelling publication bias and p‐hacking," Biometrics, The International Biometric Society, vol. 79(1), pages 319-331, March.
    16. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    17. Iwasaki, Ichiro & Satogami, Mihoko, 2023. "Gender wage gap in European emerging markets: a meta-analytic perspective," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 57, pages 1-9.
    18. Mišák, Vojtěch, 2024. "Does heat cause homicides? A meta-analysis," International Review of Law and Economics, Elsevier, vol. 78(C).
    19. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    20. Tomas Havranek & Zuzana Irsova & Lubica Laslopova & Olesia Zeynalova, 2020. "Skilled and Unskilled Labor Are Less Substitutable than Commonly Thought," Working Papers IES 2020/29, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.

    More about this item

    Keywords

    Meta-analysis; Estimator performance; Publication bias; Simulation design; Monte Carlo; Experiments;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:20/10. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.