IDEAS home Printed from https://ideas.repec.org/p/zbw/esprep/273719.html
   My bibliography  Save this paper

Meta-Analysis of Social Science Research: A Practitioner’s Guide

Author

Listed:
  • Irsova, Zuzana
  • Doucouliagos, Hristos
  • Havranek, Tomas
  • Stanley, T. D.

Abstract

This paper provides concise, nontechnical, step-by-step guidelines on how to conduct a modern meta-analysis, especially in social sciences. We treat publication bias, p-hacking, and heterogeneity as phenomena meta-analysts must always confront. To this end, we provide concrete methodological recommendations. Meta-analysis methods have advanced notably over the last few years. Yet many meta-analyses still rely on outdated approaches, some ignoring publication bias and systematic heterogeneity. While limitations persist, recently developed techniques allow robust inference even in the face of formidable problems in the underlying empirical literature. The purpose of this paper is to summarize the state of the art in a way accessible to aspiring meta-analysts in any field. We also discuss how meta-analysts can use advances in artificial intelligence to work more efficiently.

Suggested Citation

  • Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
  • Handle: RePEc:zbw:esprep:273719
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/273719/1/guidelines.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    2. Koop, Gary & Leon-Gonzalez, Roberto & Strachan, Rodney, 2012. "Bayesian model averaging in the instrumental variable regression model," Journal of Econometrics, Elsevier, vol. 171(2), pages 237-250.
    3. Fabo, Brian & Jančoková, Martina & Kempf, Elisabeth & Pástor, Ľuboš, 2021. "Fifty shades of QE: Comparing findings of central bankers and academics," Journal of Monetary Economics, Elsevier, vol. 120(C), pages 1-20.
    4. Shushu Li & Jinglan Zhang & Yong Ma, 2015. "Financial Development, Environmental Quality and Economic Growth," Sustainability, MDPI, vol. 7(7), pages 1-22, July.
    5. Amanda Kvarven & Eirik Strømland & Magnus Johannesson, 2020. "Comparing meta-analyses and preregistered multiple-laboratory replication projects," Nature Human Behaviour, Nature, vol. 4(4), pages 423-434, April.
    6. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    7. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    8. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    9. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    10. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    11. Theo S. Eicher & Chris Papageorgiou & Adrian E. Raftery, 2011. "Default priors and predictive performance in Bayesian model averaging, with application to growth determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(1), pages 30-55, January/F.
    12. Petra Valickova & Tomas Havranek & Roman Horvath, 2015. "Financial Development And Economic Growth: A Meta-Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 29(3), pages 506-526, July.
    13. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    14. Tomas Havranek & Zuzana Irsova, 2017. "Do Borders Really Slash Trade? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 65(2), pages 365-396, June.
    15. Carina Neisser, 2021. "The Elasticity of Taxable Income: A Meta-Regression Analysis [The top 1% in international and historical perspective]," The Economic Journal, Royal Economic Society, vol. 131(640), pages 3365-3391.
    16. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta‐Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.
    17. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    18. Tomas Havranek & Marek Rusnak, 2013. "Transmission Lags of Monetary Policy: A Meta-Analysis," International Journal of Central Banking, International Journal of Central Banking, vol. 9(4), pages 39-76, December.
    19. Shahram M. Amini & Christopher F. Parmeter, 2012. "Comparison Of Model Averaging Techniques: Assessing Growth Determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(5), pages 870-876, August.
    20. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    21. Doucouliagos, Hristos (Chris), 2011. "How large is large? Preliminary and relative guidelines for interpreting partial correlations in economics," Working Papers eco_2011_5, Deakin University, Department of Economics.
    22. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    23. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    24. Havranek, Tomas & Opatrny, Matej & Irsova, Zuzana & Scasny, Milan, 2023. "Publication Bias and Model Uncertainty in Measuring the Effect of Class Size on Achievement," CEPR Discussion Papers 18159, C.E.P.R. Discussion Papers.
    25. Neal S Young & John P A Ioannidis & Omar Al-Ubaydli, 2008. "Why Current Publication Practices May Distort Science," PLOS Medicine, Public Library of Science, vol. 5(10), pages 1-5, October.
    26. Timothy Cogley & Boyan Jovanovic, 2022. "Structural Breaks in an Endogenous Growth Model [Monetary Policy Regimes and Beliefs]," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 89(2), pages 666-694.
    27. Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
    28. Costa-Font, Joan & McGuire, Alistair & Stanley, Tom, 2013. "Publication selection in health policy research: The winner's curse hypothesis," Health Policy, Elsevier, vol. 109(1), pages 78-87.
    29. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    30. Strachan, Rodney W. & Inder, Brett, 2004. "Bayesian analysis of the error correction model," Journal of Econometrics, Elsevier, vol. 123(2), pages 307-325, December.
    31. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    32. Jessica Gurevitch & Julia Koricheva & Shinichi Nakagawa & Gavin Stewart, 2018. "Meta-analysis and the science of research synthesis," Nature, Nature, vol. 555(7695), pages 175-182, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    2. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    3. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    4. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    5. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    6. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    7. Bajzik, Josef & Havranek, Tomas & Irsova, Zuzana & Schwarz, Jiri, 2020. "Estimating the Armington elasticity: The importance of study design and publication bias," Journal of International Economics, Elsevier, vol. 127(C).
    8. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    9. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    10. Bajzik, Josef, 2021. "Trading volume and stock returns: A meta-analysis," International Review of Financial Analysis, Elsevier, vol. 78(C).
    11. Tomas Havranek & Anna Sokolova, 2020. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 144 Studies Say 'Probably Not'," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 35, pages 97-122, January.
    12. Simona Malovana & Martin Hodula & Zuzana Gric & Josef Bajzik, 2022. "Borrower-Based Macroprudential Measures and Credit Growth: How Biased is the Existing Literature?," Working Papers 2022/8, Czech National Bank.
    13. Zhuanlan Sun & Demi Zhu, 2023. "Investigating environmental regulation effects on technological innovation: A meta-regression analysis," Energy & Environment, , vol. 34(3), pages 463-492, May.
    14. Gric, Zuzana & Bajzík, Josef & Badura, Ondřej, 2023. "Does sentiment affect stock returns? A meta-analysis across survey-based measures," International Review of Financial Analysis, Elsevier, vol. 89(C).
    15. Anton Astakhov & Tomas Havranek & Jiri Novak, 2019. "Firm Size And Stock Returns: A Quantitative Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1463-1492, December.
    16. Xindong Xue & Mingmei Cheng & Wangyongxin Zhang, 2021. "Does Education Really Improve Health? A Meta‐Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 35(1), pages 71-105, February.
    17. Dominika Ehrenbergerova & Josef Bajzik, 2020. "The Effect of Monetary Policy on House Prices - How Strong is the Transmission?," Working Papers 2020/14, Czech National Bank.
    18. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2019. "Remittances and Economic Growth: A Quantitative Survey," EconStor Preprints 205812, ZBW - Leibniz Information Centre for Economics.
    19. Dominika Reckova & Zuzana Irsova, 2015. "Publication Bias in Measuring Anthropogenic Climate Change," Energy & Environment, , vol. 26(5), pages 853-862, September.
    20. Anna Sokolova & Todd Sorensen, 2021. "Monopsony in Labor Markets: A Meta-Analysis," ILR Review, Cornell University, ILR School, vol. 74(1), pages 27-55, January.

    More about this item

    Keywords

    meta-analysis; publication bias; p-hacking; artificial intelligence; model uncertainty;
    All these keywords.

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • B49 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Other
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Meta-Research in Economics

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:esprep:273719. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.