IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/14-22.html
   My bibliography  Save this paper

A Monte Carlo Analysis of Alternative Meta-Analysis Estimators in the Presence of Publication Bias

Author

Listed:

Abstract

This study uses Monte Carlo analysis to investigate the performances of five different meta-analysis (MA) estimators: the Fixed Effects (FE) estimator, the Weighted Least Squares (WLS) estimator, the Random Effects (RE) estimator, the Precision Effect Test (PET) estimator, and the Precision Effect Estimate with Standard Errors (PEESE) estimator. We consider two types of publication bias: publication bias directed against statistically insignificant estimates, and publication bias directed against wrong-signed estimates. Finally, we consider three cases concerning the distribution of the “true effect”: the Fixed Effects case, where there is only estimate per study, and all studies have the same true effect; the Random Effects case, where there is only one estimate per study, and there is heterogeneity in true effects across studies; and the Panel Random Effects case, where studies have multiple estimates, and true effects are random both across and within studies. Our simulations produce a number of findings that challenge results from previous research.

Suggested Citation

  • W. Robert Reed & Raymond J.G.M. Florax & Jacques Poot, 2014. "A Monte Carlo Analysis of Alternative Meta-Analysis Estimators in the Presence of Publication Bias," Working Papers in Economics 14/22, University of Canterbury, Department of Economics and Finance.
    • Handle: RePEc:cbt:econwp:14/22
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/1422.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bellavance, Franois & Dionne, Georges & Lebeau, Martin, 2009. "The value of a statistical life: A meta-analysis with a mixed effects regression model," Journal of Health Economics, Elsevier, vol. 28(2), pages 444-464, March.
    2. Mark Koetse & Raymond Florax & Henri Groot, 2010. "Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 19(2), pages 217-236, June.
    3. Jasper M. Dalhuisen & Raymond J. G. M. Florax & JHenri L. F. de Groot & Peter Nijkamp, 2003. "Price and Income Elasticities of Residential Water Demand: A Meta-Analysis," Land Economics, University of Wisconsin Press, vol. 79(2), pages 292-308.
    4. Hristos Doucouliagos & Martin Paldam, 2009. "The Aid Effectiveness Literature: The Sad Results Of 40 Years Of Research," Journal of Economic Surveys, Wiley Blackwell, vol. 23(3), pages 433-461, July.
    5. Hristos Doucouliagos & Janto Haman & T.D. Stanley, 2012. "Pay for Performance and Corporate Governance Reform," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 51(3), pages 670-703, July.
    6. Doucouliagos, Chris & Stanley, T.D. & Giles, Margaret, 2012. "Are estimates of the value of a statistical life exaggerated?," Journal of Health Economics, Elsevier, vol. 31(1), pages 197-206.
    7. Stanley, T. D. & Doucouliagos, Hristos, 2013. "Better than random: weighted least squares meta-regression analysis," Working Papers eco_2013_2, Deakin University, Department of Economics.
    8. Hristos Doucouliagos & Martin Paldam, 2013. "The Robust Result in Meta-analysis of Aid Effectiveness: A Response to Mekasha and Tarp," Journal of Development Studies, Taylor & Francis Journals, vol. 49(4), pages 584-587, April.
    9. Tseday Jemaneh Mekasha & Finn Tarp, 2013. "Aid and Growth: What Meta-Analysis Reveals," Journal of Development Studies, Taylor & Francis Journals, vol. 49(4), pages 564-583, April.
    10. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    11. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    12. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    13. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    14. Nelson, Jon P., 2014. "Estimating the price elasticity of beer: Meta-analysis of data with heterogeneity, dependence, and publication bias," Journal of Health Economics, Elsevier, vol. 33(C), pages 180-187.
    15. Tomas Havranek & Zuzana Irsova, 2017. "Do Borders Really Slash Trade? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 65(2), pages 365-396, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paldam, Martin, 2015. "Meta-analysis in a nutshell: Techniques and general findings," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-14.
    2. Nazila Alinaghi & W. Robert Reed, 2021. "Taxes and Economic Growth in OECD Countries: A Meta-analysis," Public Finance Review, , vol. 49(1), pages 3-40, January.
    3. Randolph Luca Bruno & Maria Cipollina, 2018. "A meta†analysis of the indirect impact of foreign direct investment in old and new EU member states: Understanding productivity spillovers," The World Economy, Wiley Blackwell, vol. 41(5), pages 1342-1377, May.
    4. Nazila Alinaghi & W. Robert Reed, 2016. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 16/26, University of Canterbury, Department of Economics and Finance.
    5. Petr Polak & Nikol Polakova & Anna Tlusta, 2020. "How Bad Are Trade Wars? Evidence from Tariffs," Working Papers 2020/15, Czech National Bank.
    6. Nazila Alinaghi & W. Robert Reed, 2016. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 16/26, University of Canterbury, Department of Economics and Finance.
    7. Gunby, Philip & Jin, Yinghua & Robert Reed, W., 2017. "Did FDI Really Cause Chinese Economic Growth? A Meta-Analysis," World Development, Elsevier, vol. 90(C), pages 242-255.
    8. Jianhua Duan & Kuntal K. Das & Laura Meriluoto & W. Robert Reed, 2019. "Spillovers and Exports: A Meta-Analysis," Working Papers in Economics 19/19, University of Canterbury, Department of Economics and Finance.
    9. Hippolyte W. BALIMA & Eric Gabin KILAMA & René TAPSOBA, 2017. "Settling the Inflation Targeting Debate: Lights from a Meta-Regression Analysis," Working Papers 4080, FERDI.
    10. Xue, Xindong & Reed, W. Robert & Menclova, Andrea, 2020. "Social capital and health: a meta-analysis," Journal of Health Economics, Elsevier, vol. 72(C).
    11. Diana Zigraiova & Tomas Havranek, 2016. "Bank Competition And Financial Stability: Much Ado About Nothing?," Journal of Economic Surveys, Wiley Blackwell, vol. 30(5), pages 944-981, December.
    12. Sanghyun Hong & W. Robert Reed, 2019. "Towards an Experimental Framework for Assessing Meta-Analysis Methods, with a Focus on Andrews-Kasy Estimators," Working Papers in Economics 19/13, University of Canterbury, Department of Economics and Finance.
    13. Martin Paldam, 2016. "Simulating an empirical paper by the rational economist," Empirical Economics, Springer, vol. 50(4), pages 1383-1407, June.
    14. Sanghyun Hong & W. Robert Reed, 2019. "A Performance Analysis of Some New Meta-Analysis Estimators Designed to Correct Publication Bias," Working Papers in Economics 19/04, University of Canterbury, Department of Economics and Finance.
    15. Jianhua Duan & Kuntal K. Das & Laura Meriluoto & W. Robert Reed, 2020. "Estimating the effect of spillovers on exports: a meta-analysis," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 156(2), pages 219-249, May.
    16. Nelson, Jon Paul, 2020. "Fixed-effect versus random-effects meta-analysis in economics: A study of pass-through rates for alcohol beverage excise taxes," Economics Discussion Papers 2020-1, Kiel Institute for the World Economy (IfW Kiel).
    17. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    18. Jurkat, Anne & Klump, Rainer & Schneider, Florian, 2023. "Robots and Wages: A Meta-Analysis," EconStor Preprints 274156, ZBW - Leibniz Information Centre for Economics.
    19. Sanghyun Hong & W. Robert Reed, 2020. "Using Monte Carlo Experiments to Select Meta-Analytic Estimators," Working Papers in Economics 20/10, University of Canterbury, Department of Economics and Finance.
    20. Balima, Hippolyte W. & Kilama, Eric G. & Tapsoba, René, 2020. "Inflation targeting: Genuine effects or publication selection bias?," European Economic Review, Elsevier, vol. 128(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:wly:econjl:v::y:2017:i:605:p:f236-f265 is not listed on IDEAS
    2. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    3. Nazila Alinaghi & W. Robert Reed, 2021. "Taxes and Economic Growth in OECD Countries: A Meta-analysis," Public Finance Review, , vol. 49(1), pages 3-40, January.
    4. Nazila Alinaghi & W. Robert Reed, 2016. "Meta-Analysis and Publication Bias: How Well Does the FAT-PET-PEESE Procedure Work?," Working Papers in Economics 16/26, University of Canterbury, Department of Economics and Finance.
    5. Stanley, T. D. & Doucouliagos, Hristos, 2013. "Better than random: weighted least squares meta-regression analysis," Working Papers eco_2013_2, Deakin University, Department of Economics.
    6. Doucouliagos, Hristos & Stanley, T.D. & Viscusi, W. Kip, 2014. "Publication selection and the income elasticity of the value of a statistical life," Journal of Health Economics, Elsevier, vol. 33(C), pages 67-75.
    7. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    8. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    9. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    10. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    11. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    12. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    13. Stanley, T. D. & Doucouliagos, Hristos, 2011. "Meta-regression approximations to reduce publication selection bias," Working Papers eco_2011_4, Deakin University, Department of Economics.
    14. Geyer-Klingeberg, Jerome & Hang, Markus & Rathgeber, Andreas W., 2019. "What drives financial hedging? A meta-regression analysis of corporate hedging determinants," International Review of Financial Analysis, Elsevier, vol. 61(C), pages 203-221.
    15. Doucouliagos, Chris & Stanley, T.D. & Giles, Margaret, 2012. "Are estimates of the value of a statistical life exaggerated?," Journal of Health Economics, Elsevier, vol. 31(1), pages 197-206.
    16. Diana Zigraiova & Tomas Havranek, 2016. "Bank Competition And Financial Stability: Much Ado About Nothing?," Journal of Economic Surveys, Wiley Blackwell, vol. 30(5), pages 944-981, December.
    17. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    18. Balima, Hippolyte W. & Sokolova, Anna, 2021. "IMF programs and economic growth: A meta-analysis," Journal of Development Economics, Elsevier, vol. 153(C).
    19. Sefa Awaworyi Churchill & Vinod Mishra, 2018. "Returns to education in China: a meta-analysis," Applied Economics, Taylor & Francis Journals, vol. 50(54), pages 5903-5919, November.
    20. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    21. Liu, Boying & Richard Shumway, C., 2016. "Substitution elasticities between GHG-polluting and nonpolluting inputs in agricultural production: A meta-regression," Energy Economics, Elsevier, vol. 54(C), pages 123-132.

    More about this item

    Keywords

    Meta-analysis; Random effects; Fixed effects; publication bias; Monte Carlo; Simulations;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:14/22. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.