IDEAS home Printed from https://ideas.repec.org/p/fau/wpaper/wp2023_25.html
   My bibliography  Save this paper

Meta-Analysis of Social Science Research: A Practitioner´s Guide

Author

Listed:
  • Zuzana Irsova

    (Charles University, Prague & Anglo-American University, Prague)

  • Hristos Doucouliagos

    (Department of Economics and Deakin Laboratory for the Meta-Analysis of Research. Deakin University, Melbourne, Australia.)

  • Tomas Havranek

    (Charles University, Prague & Centre for Economic Policy Research, London)

  • T. D. Stanley

    (4Department of Economics and Deakin Laboratory for the Meta-Analysis of Research, Deakin University, Melbourne, Australia)

Abstract

This paper provides concise, nontechnical, step-by-step guidelines on how to conduct a modern meta-analysis, especially in social sciences. We treat publication bias, p-hacking, and heterogeneity as phenomena meta-analysts must always confront. To this end, we provide concrete methodological recommendations. Meta-analysis methods have advanced notably over the last few years. Yet many meta-analyses still rely on outdated approaches, some ignoring publication bias and systematic heterogeneity. While limitations persist, recently developed techniques allow robust inference even in the face of formidable problems in the underlying empirical literature. The purpose of this paper is to summarize the state of the art in a way accessible to aspiring meta-analysts in any field. We also discuss how meta-analysts can use advances in artificial intelligence to work more efficiently.

Suggested Citation

  • Zuzana Irsova & Hristos Doucouliagos & Tomas Havranek & T. D. Stanley, 2023. "Meta-Analysis of Social Science Research: A Practitioner´s Guide," Working Papers IES 2023/25, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2023.
  • Handle: RePEc:fau:wpaper:wp2023_25
    as

    Download full text from publisher

    File URL: https://ies.fsv.cuni.cz/en/meta-analysis-social-science-research-practitioners-guide
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Cala, Petr & Havranek, Tomas & Irsova, Zuzana & Matousek, Jindrich & Novak, Jiri, 2022. "Financial Incentives and Performance: A Meta-Analysis of Economics Evidence," CEPR Discussion Papers 17680, C.E.P.R. Discussion Papers.
    2. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    3. Koop, Gary & Leon-Gonzalez, Roberto & Strachan, Rodney, 2012. "Bayesian model averaging in the instrumental variable regression model," Journal of Econometrics, Elsevier, vol. 171(2), pages 237-250.
    4. Matej Opatrny & Tomas Havranek & Zuzana Irsova & Milan Scasny, 2023. "Publication Bias and Model Uncertainty in Measuring the Effect of Class Size on Achievement," Working Papers IES 2023/19, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised May 2023.
    5. Tomas Havranek & Dominik Herman & Zuzana Irsova, 2018. "Does Daylight Saving Save Electricity? A Meta-Analysis," The Energy Journal, , vol. 39(2), pages 35-61, March.
    6. Fabo, Brian & Jančoková, Martina & Kempf, Elisabeth & Pástor, Ľuboš, 2021. "Fifty shades of QE: Comparing findings of central bankers and academics," Journal of Monetary Economics, Elsevier, vol. 120(C), pages 1-20.
    7. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    8. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    9. Shushu Li & Jinglan Zhang & Yong Ma, 2015. "Financial Development, Environmental Quality and Economic Growth," Sustainability, MDPI, vol. 7(7), pages 1-22, July.
    10. Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
    11. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    12. Havranek, Tomas & Horvath, Roman & Irsova, Zuzana & Rusnak, Marek, 2015. "Cross-country heterogeneity in intertemporal substitution," Journal of International Economics, Elsevier, vol. 96(1), pages 100-118.
    13. Amanda Kvarven & Eirik Strømland & Magnus Johannesson, 2020. "Comparing meta-analyses and preregistered multiple-laboratory replication projects," Nature Human Behaviour, Nature, vol. 4(4), pages 423-434, April.
    14. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    15. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    16. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    17. Bajzik, Josef & Havranek, Tomas & Irsova, Zuzana & Schwarz, Jiri, 2020. "Estimating the Armington elasticity: The importance of study design and publication bias," Journal of International Economics, Elsevier, vol. 127(C).
    18. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    19. Tomas Havranek & Marek Rusnak, 2013. "Transmission Lags of Monetary Policy: A Meta-Analysis," International Journal of Central Banking, International Journal of Central Banking, vol. 9(4), pages 39-76, December.
    20. Costa-Font, Joan & McGuire, Alistair & Stanley, Tom, 2013. "Publication selection in health policy research: The winner's curse hypothesis," Health Policy, Elsevier, vol. 109(1), pages 78-87.
    21. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    22. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    23. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    24. Stanley, T. D. & Jarrell, Stephen B. & Doucouliagos, Hristos, 2010. "Could It Be Better to Discard 90% of the Data? A Statistical Paradox," The American Statistician, American Statistical Association, vol. 64(1), pages 70-77.
    25. Kroupova, Katerina & Havranek, Tomas & Irsova, Zuzana, 2024. "Student Employment and Education: A Meta-Analysis," Economics of Education Review, Elsevier, vol. 100(C).
    26. Havranek, Tomas & Irsova, Zuzana & Janda, Karel & Zilberman, David, 2015. "Selective reporting and the social cost of carbon," Energy Economics, Elsevier, vol. 51(C), pages 394-406.
    27. T. D. Stanley & Hristos Doucouliagos & Tomas Havranek, 2023. "Meta-Analyses of Partial Correlations Are Biased: Detection and Solutions," Working Papers IES 2023/17, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised May 2023.
    28. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    29. Strachan, Rodney W. & Inder, Brett, 2004. "Bayesian analysis of the error correction model," Journal of Econometrics, Elsevier, vol. 123(2), pages 307-325, December.
    30. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    31. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    32. Petra Valickova & Tomas Havranek & Roman Horvath, 2015. "Financial Development And Economic Growth: A Meta-Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 29(3), pages 506-526, July.
    33. George-Marios Angeletos & Zhen Huo, 2021. "Myopia and Anchoring," American Economic Review, American Economic Association, vol. 111(4), pages 1166-1200, April.
    34. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    35. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    36. Theo S. Eicher & Chris Papageorgiou & Adrian E. Raftery, 2011. "Default priors and predictive performance in Bayesian model averaging, with application to growth determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(1), pages 30-55, January/F.
    37. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LP, vol. 19(1), pages 4-60, March.
    38. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    39. Elminejad, Ali & Havranek, Tomas & Irsova, Zuzana, 2022. "Relative Risk Aversion: A Meta-Analysis," MetaArXiv b8uhe, Center for Open Science.
    40. Tomas Havranek & Zuzana Irsova, 2017. "Do Borders Really Slash Trade? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 65(2), pages 365-396, June.
    41. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    42. Carina Neisser, 2021. "The Elasticity of Taxable Income: A Meta-Regression Analysis [The top 1% in international and historical perspective]," The Economic Journal, Royal Economic Society, vol. 131(640), pages 3365-3391.
    43. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    44. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    45. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta‐Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.
    46. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    47. Shahram M. Amini & Christopher F. Parmeter, 2012. "Comparison Of Model Averaging Techniques: Assessing Growth Determinants," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(5), pages 870-876, August.
    48. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    49. Doucouliagos, Hristos (Chris), 2011. "How large is large? Preliminary and relative guidelines for interpreting partial correlations in economics," Working Papers eco_2011_5, Deakin University, Department of Economics.
    50. Tomas Havranek & Zuzana Irsova & Tomas Vlach, 2018. "Measuring the Income Elasticity of Water Demand: The Importance of Publication and Endogeneity Biases," Land Economics, University of Wisconsin Press, vol. 94(2), pages 259-283.
    51. Diana Zigraiova & Tomas Havranek, 2016. "Bank Competition And Financial Stability: Much Ado About Nothing?," Journal of Economic Surveys, Wiley Blackwell, vol. 30(5), pages 944-981, December.
    52. Tomas Havranek & Zuzana Irsova & Olesia Zeynalova, 2018. "Tuition Fees and University Enrolment: A Meta‐Regression Analysis," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 80(6), pages 1145-1184, December.
    53. Mohammed Ziaur Rehman & Nasir Ali & Najeeb Muhammad Nasir, 2015. "Financial Development, Savings and Economic Growth: Evidence from Bahrain Using VAR," International Journal of Financial Research, International Journal of Financial Research, Sciedu Press, vol. 6(2), pages 112-123, April.
    54. Jessica Gurevitch & Julia Koricheva & Shinichi Nakagawa & Gavin Stewart, 2018. "Meta-analysis and the science of research synthesis," Nature, Nature, vol. 555(7695), pages 175-182, March.
    55. Neal S Young & John P A Ioannidis & Omar Al-Ubaydli, 2008. "Why Current Publication Practices May Distort Science," PLOS Medicine, Public Library of Science, vol. 5(10), pages 1-5, October.
    56. Tomas Havranek & Zuzana Irsova & Lubica Laslopova & Olesia Zeynalova, 2024. "Publication and Attenuation Biases in Measuring Skill Substitution," The Review of Economics and Statistics, MIT Press, vol. 106(5), pages 1187-1200, September.
    57. Timothy Cogley & Boyan Jovanovic, 2022. "Structural Breaks in an Endogenous Growth Model," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 89(2), pages 666-694.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    2. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    3. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    4. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    5. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    6. Bajzik, Josef & Havranek, Tomas & Irsova, Zuzana & Schwarz, Jiri, 2020. "Estimating the Armington elasticity: The importance of study design and publication bias," Journal of International Economics, Elsevier, vol. 127(C).
    7. Sebastian Gechert & Tomas Havranek & Zuzana Irsova & Dominika Kolcunova, 2022. "Measuring Capital-Labor Substitution: The Importance of Method Choices and Publication Bias," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 45, pages 55-82, July.
    8. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    9. Bajzik, Josef, 2021. "Trading volume and stock returns: A meta-analysis," International Review of Financial Analysis, Elsevier, vol. 78(C).
    10. Tomas Havranek & Anna Sokolova, 2020. "Do Consumers Really Follow a Rule of Thumb? Three Thousand Estimates from 144 Studies Say 'Probably Not'," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 35, pages 97-122, January.
    11. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    12. Sintos, Andreas, 2023. "Does inflation worsen income inequality? A meta-analysis," Economic Systems, Elsevier, vol. 47(4).
    13. Simona Malovana & Martin Hodula & Zuzana Gric & Josef Bajzik, 2022. "Borrower-Based Macroprudential Measures and Credit Growth: How Biased is the Existing Literature?," Working Papers 2022/8, Czech National Bank.
    14. Gric, Zuzana & Bajzík, Josef & Badura, Ondřej, 2023. "Does sentiment affect stock returns? A meta-analysis across survey-based measures," International Review of Financial Analysis, Elsevier, vol. 89(C).
    15. Slichter, David & Tran, Nhan, 2023. "Do better journals publish better estimates?," MPRA Paper 118433, University Library of Munich, Germany.
    16. Josef Bajzik & Tomas Havranek & Zuzana Irsova & Jiri Novak, 2023. "Do Shareholder Activism Announcements Affect Stock Prices? A Meta-Analysis," Working Papers 2023/17, Czech National Bank.
    17. Anton Astakhov & Tomas Havranek & Jiri Novak, 2019. "Firm Size And Stock Returns: A Quantitative Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1463-1492, December.
    18. Matilde Cardoso & Pedro Cunha Neves & Oscar Afonso & Elena Sochirca, 2021. "The effects of offshoring on wages: a meta-analysis," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 157(1), pages 149-179, February.
    19. Mojmir Hampl & Tomas Havranek, 2020. "Central Bank Equity as an Instrument of Monetary Policy," Comparative Economic Studies, Palgrave Macmillan;Association for Comparative Economic Studies, vol. 62(1), pages 49-68, March.
    20. Katarina Gomoryova, 2024. "Female Leadership and Financial Performance: A Meta-Analysis," Working Papers IES 2024/6, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Jan 2024.

    More about this item

    Keywords

    meta-analysis; publication bias; p-hacking; artificial intelligence; model uncertainty;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fau:wpaper:wp2023_25. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Natalie Svarcova (email available below). General contact details of provider: https://edirc.repec.org/data/icunicz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.