IDEAS home Printed from https://ideas.repec.org/p/fau/wpaper/wp2023_25.html
   My bibliography  Save this paper

Meta-Analysis of Social Science Research: A Practitioner´s Guide

Author

Listed:
  • Zuzana Irsova

    (Charles University, Prague & Anglo-American University, Prague)

  • Hristos Doucouliagos

    (Department of Economics and Deakin Laboratory for the Meta-Analysis of Research. Deakin University, Melbourne, Australia.)

  • Tomas Havranek

    (Charles University, Prague & Centre for Economic Policy Research, London)

  • T. D. Stanley

    (4Department of Economics and Deakin Laboratory for the Meta-Analysis of Research, Deakin University, Melbourne, Australia)

Abstract

This paper provides concise, nontechnical, step-by-step guidelines on how to conduct a modern meta-analysis, especially in social sciences. We treat publication bias, p-hacking, and heterogeneity as phenomena meta-analysts must always confront. To this end, we provide concrete methodological recommendations. Meta-analysis methods have advanced notably over the last few years. Yet many meta-analyses still rely on outdated approaches, some ignoring publication bias and systematic heterogeneity. While limitations persist, recently developed techniques allow robust inference even in the face of formidable problems in the underlying empirical literature. The purpose of this paper is to summarize the state of the art in a way accessible to aspiring meta-analysts in any field. We also discuss how meta-analysts can use advances in artificial intelligence to work more efficiently.

Suggested Citation

  • Zuzana Irsova & Hristos Doucouliagos & Tomas Havranek & T. D. Stanley, 2023. "Meta-Analysis of Social Science Research: A Practitioner´s Guide," Working Papers IES 2023/25, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2023.
  • Handle: RePEc:fau:wpaper:wp2023_25
    as

    Download full text from publisher

    File URL: https://ies.fsv.cuni.cz/en/meta-analysis-social-science-research-practitioners-guide
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    2. Koop, Gary & Leon-Gonzalez, Roberto & Strachan, Rodney, 2012. "Bayesian model averaging in the instrumental variable regression model," Journal of Econometrics, Elsevier, vol. 171(2), pages 237-250.
    3. Matej Opatrny & Tomas Havranek & Zuzana Irsova & Milan Scasny, 2023. "Publication Bias and Model Uncertainty in Measuring the Effect of Class Size on Achievement," Working Papers IES 2023/19, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised May 2023.
    4. Fabo, Brian & Jančoková, Martina & Kempf, Elisabeth & Pástor, Ľuboš, 2021. "Fifty shades of QE: Comparing findings of central bankers and academics," Journal of Monetary Economics, Elsevier, vol. 120(C), pages 1-20.
    5. Shushu Li & Jinglan Zhang & Yong Ma, 2015. "Financial Development, Environmental Quality and Economic Growth," Sustainability, MDPI, vol. 7(7), pages 1-22, July.
    6. Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
    7. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    8. Amanda Kvarven & Eirik Strømland & Magnus Johannesson, 2020. "Comparing meta-analyses and preregistered multiple-laboratory replication projects," Nature Human Behaviour, Nature, vol. 4(4), pages 423-434, April.
    9. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    10. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    11. Tomas Havranek & Marek Rusnak, 2013. "Transmission Lags of Monetary Policy: A Meta-Analysis," International Journal of Central Banking, International Journal of Central Banking, vol. 9(4), pages 39-76, December.
    12. Costa-Font, Joan & McGuire, Alistair & Stanley, Tom, 2013. "Publication selection in health policy research: The winner's curse hypothesis," Health Policy, Elsevier, vol. 109(1), pages 78-87.
    13. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    14. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    15. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    16. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    17. Strachan, Rodney W. & Inder, Brett, 2004. "Bayesian analysis of the error correction model," Journal of Econometrics, Elsevier, vol. 123(2), pages 307-325, December.
    18. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    19. Petra Valickova & Tomas Havranek & Roman Horvath, 2015. "Financial Development And Economic Growth: A Meta-Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 29(3), pages 506-526, July.
    20. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    21. Theo S. Eicher & Chris Papageorgiou & Adrian E. Raftery, 2011. "Default priors and predictive performance in Bayesian model averaging, with application to growth determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(1), pages 30-55, January/F.
    22. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    23. Tomas Havranek & Zuzana Irsova, 2017. "Do Borders Really Slash Trade? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 65(2), pages 365-396, June.
    24. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    25. Carina Neisser, 2021. "The Elasticity of Taxable Income: A Meta-Regression Analysis [The top 1% in international and historical perspective]," The Economic Journal, Royal Economic Society, vol. 131(640), pages 3365-3391.
    26. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta‐Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.
    27. Shahram M. Amini & Christopher F. Parmeter, 2012. "Comparison Of Model Averaging Techniques: Assessing Growth Determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(5), pages 870-876, August.
    28. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    29. Doucouliagos, Hristos (Chris), 2011. "How large is large? Preliminary and relative guidelines for interpreting partial correlations in economics," Working Papers eco_2011_5, Deakin University, Department of Economics.
    30. Jessica Gurevitch & Julia Koricheva & Shinichi Nakagawa & Gavin Stewart, 2018. "Meta-analysis and the science of research synthesis," Nature, Nature, vol. 555(7695), pages 175-182, March.
    31. Neal S Young & John P A Ioannidis & Omar Al-Ubaydli, 2008. "Why Current Publication Practices May Distort Science," PLOS Medicine, Public Library of Science, vol. 5(10), pages 1-5, October.
    32. Timothy Cogley & Boyan Jovanovic, 2022. "Structural Breaks in an Endogenous Growth Model [Monetary Policy Regimes and Beliefs]," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 89(2), pages 666-694.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    2. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    3. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    4. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    5. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    6. Bajzik, Josef & Havranek, Tomas & Irsova, Zuzana & Schwarz, Jiri, 2020. "Estimating the Armington elasticity: The importance of study design and publication bias," Journal of International Economics, Elsevier, vol. 127(C).
    7. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    8. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    9. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    10. Bajzik, Josef, 2021. "Trading volume and stock returns: A meta-analysis," International Review of Financial Analysis, Elsevier, vol. 78(C).
    11. Tomas Havranek & Anna Sokolova, 2020. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 144 Studies Say 'Probably Not'," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 35, pages 97-122, January.
    12. Simona Malovana & Martin Hodula & Zuzana Gric & Josef Bajzik, 2022. "Borrower-Based Macroprudential Measures and Credit Growth: How Biased is the Existing Literature?," Working Papers 2022/8, Czech National Bank.
    13. Zhuanlan Sun & Demi Zhu, 2023. "Investigating environmental regulation effects on technological innovation: A meta-regression analysis," Energy & Environment, , vol. 34(3), pages 463-492, May.
    14. Gric, Zuzana & Bajzík, Josef & Badura, Ondřej, 2023. "Does sentiment affect stock returns? A meta-analysis across survey-based measures," International Review of Financial Analysis, Elsevier, vol. 89(C).
    15. Anton Astakhov & Tomas Havranek & Jiri Novak, 2019. "Firm Size And Stock Returns: A Quantitative Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1463-1492, December.
    16. Dominika Ehrenbergerova & Josef Bajzik, 2020. "The Effect of Monetary Policy on House Prices - How Strong is the Transmission?," Working Papers 2020/14, Czech National Bank.
    17. Dominika Reckova & Zuzana Irsova, 2015. "Publication Bias in Measuring Anthropogenic Climate Change," Energy & Environment, , vol. 26(5), pages 853-862, September.
    18. Awaworyi Churchill, S. & Yew, S.L., 2017. "Are government transfers harmful to economic growth? A meta-analysis," Economic Modelling, Elsevier, vol. 64(C), pages 270-287.
    19. Anna Sokolova & Todd Sorensen, 2021. "Monopsony in Labor Markets: A Meta-Analysis," ILR Review, Cornell University, ILR School, vol. 74(1), pages 27-55, January.
    20. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2019. "Remittances and Economic Growth: A Quantitative Survey," MPRA Paper 96823, University Library of Munich, Germany.

    More about this item

    Keywords

    meta-analysis; publication bias; p-hacking; artificial intelligence; model uncertainty;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fau:wpaper:wp2023_25. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Natalie Svarcova (email available below). General contact details of provider: https://edirc.repec.org/data/icunicz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.