IDEAS home Printed from https://ideas.repec.org/a/pal/jintbs/v52y2021i1d10.1057_s41267-020-00385-z.html
   My bibliography  Save this article

The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews

Author

Listed:
  • Piers Steel

    (University of Calgary)

  • Sjoerd Beugelsdijk

    (University of Groningen)

  • Herman Aguinis

    (The George Washington University)

Abstract

Meta-analyses summarize a field’s research base and are therefore highly influential. Despite their value, the standards for an excellent meta-analysis, one that is potentially award-winning, have changed in the last decade. Each step of a meta-analysis is now more formalized, from the identification of relevant articles to coding, moderator analysis, and reporting of results. What was exemplary a decade ago can be somewhat dated today. Using the award-winning meta-analysis by Stahl et al. (Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies, 41(4):690–709, 2010) as an exemplar, we adopted a multi-disciplinary approach (e.g., management, psychology, health sciences) to summarize the anatomy (i.e., fundamental components) of a modern meta-analysis, focusing on: (1) data collection (i.e., literature search and screening, coding), (2) data preparation (i.e., treatment of multiple effect sizes, outlier identification and management, publication bias), (3) data analysis (i.e., average effect sizes, heterogeneity of effect sizes, moderator search), and (4) reporting (i.e., transparency and reproducibility, future research directions). In addition, we provide guidelines and a decision-making tree for when even foundational and highly cited meta-analyses should be updated. Based on the latest evidence, we summarize what journal editors and reviewers should expect, authors should provide, and readers (i.e., other researchers, practitioners, and policymakers) should consider about meta-analytic reviews.

Suggested Citation

  • Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    • Handle: RePEc:pal:jintbs:v:52:y:2021:i:1:d:10.1057_s41267-020-00385-z
    DOI: 10.1057/s41267-020-00385-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41267-020-00385-z
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41267-020-00385-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bowen, Frances E. & Rostami, Mahdi & Steel, Piers, 2010. "Timing is everything: A meta-analysis of the relationships between organizational performance and innovation," Journal of Business Research, Elsevier, vol. 63(11), pages 1179-1185, November.
    2. Steel, Piers & Johnson, Jeff W. & Jeanneret, P. Richard & Scherbaum, Charles A. & Hoffman, Calvin C. & Foster, Jeff, 2010. "At Sea With Synthetic Validity," Industrial and Organizational Psychology, Cambridge University Press, vol. 3(3), pages 371-383, September.
    3. Robbert Maseland & Douglas Dow & Piers Steel, 2018. "The Kogut and Singh national cultural distance index: Time to start using it as a springboard rather than a crutch," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1154-1166, December.
    4. Tatiana Kostova & Sjoerd Beugelsdijk & W. Richard Scott & Vincent E. Kunst & Chei Hwee Chua & Marc Essen, 2020. "The construct of institutional distance through the lens of different institutional perspectives: Review, analysis, and recommendations," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(4), pages 467-497, June.
    5. Sjoerd Beugelsdijk & Arjen Witteloostuijn & Klaus E. Meyer, 2020. "A new approach to data access and research transparency (DART)," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(6), pages 887-905, August.
    6. Klaus E Meyer & Arjen Witteloostuijn & Sjoerd Beugelsdijk, 2017. "What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 48(5), pages 535-551, July.
    7. Huw T. O. Davies & Sandra M. Nutley & Peter C. Smith, 1999. "Viewpoint: Editorial: What Works? The Role of Evidence in Public Sector Policy and Practice," Public Money & Management, Taylor & Francis Journals, vol. 19(1), pages 3-5, January.
    8. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    9. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    10. Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    11. Aguinis, Herman & Banks, George C. & Rogelberg, Steven G. & Cascio, Wayne F., 2020. "Actionable recommendations for narrowing the science-practice gap in open science," Organizational Behavior and Human Decision Processes, Elsevier, vol. 158(C), pages 27-35.
    12. Johnson, Blair T. & Hennessy, Emily A., 2019. "Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses," Social Science & Medicine, Elsevier, vol. 233(C), pages 237-251.
    13. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    14. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    15. Vas Taras & Piers Steel, 2009. "Beyond Hofstede: Challenging the Ten Commandments of Cross-Cultural Research," Palgrave Macmillan Books, in: Cheryl Nakata (ed.), Beyond Hofstede, chapter 3, pages 40-60, Palgrave Macmillan.
    16. Sjoerd Beugelsdijk & Björn Ambos & Phillip C Nell, 2018. "Conceptualizing and measuring distance in international business research: Recurring questions and best practice guidelines," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1113-1137, December.
    17. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    18. Tomáš Havránek & T. D. Stanley & Hristos Doucouliagos & Pedro Bom & Jerome Geyer‐Klingeberg & Ichiro Iwasaki & W. Robert Reed & Katja Rost & R. C. M. van Aert, 2020. "Reporting Guidelines For Meta‐Analysis In Economics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(3), pages 469-475, July.
    19. Günter K Stahl & Martha L Maznevski & Andreas Voigt & Karsten Jonsen, 2010. "Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 41(4), pages 690-709, May.
    20. Sjoerd Beugelsdijk & Tatiana Kostova & Vincent E. Kunst & Ettore Spadafora & Marc van Essen, 2018. "Cultural Distance and Firm Internationalization," Post-Print hal-02312065, HAL.
    21. Donald D. Bergh & Herman Aguinis & Ciaran Heavey & David J. Ketchen & Brian K. Boyd & Peiran Su & Cubie L. L. Lau & Harry Joo, 2016. "Using meta-analytic structural equation modeling to advance strategic management research: Guidelines and an empirical illustration via the strategic leadership-performance relationship," Strategic Management Journal, Wiley Blackwell, vol. 37(3), pages 477-497, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas Rockstuhl & Dongyuan Wu & James H. Dulebohn & Chenwei Liao & Julia E. Hoch, 2023. "Cultural congruence or compensation? A meta-analytic test of transformational and transactional leadership effects across cultures," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 54(3), pages 476-504, April.
    2. Justyna Wiktorowicz & Izabela Warwas & Dariusz Turek & Iwa Kuchciak, 2022. "Does generativity matter? A meta-analysis on individual work outcomes," European Journal of Ageing, Springer, vol. 19(4), pages 977-995, December.
    3. Luis Alfonso Dau & Grazia D. Santangelo & Arjen Witteloostuijn, 2022. "Replication studies in international business," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(2), pages 215-230, March.
    4. Anil Nair & Mehdi Sharifi Khobdeh & Aydin Oksoy & Orhun Guldiken & Chris H. Willis, 2023. "A review of strategic management research on India," Asia Pacific Journal of Management, Springer, vol. 40(4), pages 1341-1392, December.
    5. Josef Bajzik, 2023. "Does Shareholder Activism Have a Long-Lasting Impact on Company Value? A Meta-Analysis," Working Papers 2023/10, Czech National Bank.
    6. Angélica Pigola & Priscila Rezende Costa, 2022. "In search of understanding about knowledge and learning on innovation performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3995-4022, July.
    7. Mackey, Jeremy D., 2022. "The effect of cultural values on the strength of the relationship between interpersonal and organizational workplace deviance," Journal of Business Research, Elsevier, vol. 149(C), pages 760-771.
    8. Christopher Hansen & Holger Steinmetz & Jörn Block, 2022. "How to conduct a meta-analysis in eight steps: a practical guide," Management Review Quarterly, Springer, vol. 72(1), pages 1-19, February.
    9. Davila, Jessenia & Duran, Patricio & Gómez-Mejía, Luis & Sanchez-Bueno, Maria J., 2023. "Socioemotional wealth and family firm performance: A meta-analytic integration," Journal of Family Business Strategy, Elsevier, vol. 14(2).
    10. Luiz Antonio de Camargo Guerrazzi & Fernando Antonio Ribeiro Serra & Manuel Portugal Ferreira & Vanessa Vasconcelos Scazziota, 2022. "Using Meta-Analytic Structural Equation Modelling to Advance Entrepreneurship Research: A Study on the Liabilities of Newness and Smallness," Journal of Entrepreneurship and Innovation in Emerging Economies, Entrepreneurship Development Institute of India, vol. 31(3), pages 603-631, November.
    11. Xie, Qiuhao & Gao, Ying & Xia, Nini & Zhang, Shuibo & Tao, Guowu, 2023. "Coopetition and organizational performance outcomes: A meta-analysis of the main and moderator effects," Journal of Business Research, Elsevier, vol. 154(C).
    12. Sasa Ding & Frank McDonald & Yingqi Wei, 2021. "Is Internationalization Beneficial to Innovation? Evidence from a Meta-analysis," Management International Review, Springer, vol. 61(4), pages 469-519, August.
    13. Joern Block & Christopher Hansen & Holger Steinmetz, 2023. "Are Family Firms Doing More Innovation Output With Less Innovation Input? A Replication and Extension," Entrepreneurship Theory and Practice, , vol. 47(4), pages 1496-1520, July.
    14. Alvaro Cuervo-Cazurra & Patricio Duran & Jean-Luc Arrègle & Marc van Essen, 2023. "Host Country Politics and Internationalization," Post-Print hal-04381312, HAL.
    15. Li, Lingjia & Shan, Shuo & Dai, Jing & Che, Wen & Shou, Yongyi, 2022. "The impact of green supply chain management on green innovation: A meta-analysis from the inter-organizational learning perspective," International Journal of Production Economics, Elsevier, vol. 250(C).
    16. Chenxi Wan & Carlos M. P. Sousa & Jorge Lengler & Qun Tan, 2023. "Entry Mode Choice: A Meta-Analysis of Antecedents and Outcomes," Management International Review, Springer, vol. 63(2), pages 193-246, April.
    17. Bommer, William H. & Milevoj, Emil & Rana, Shailesh, 2023. "The intention to use cryptocurrency: A meta-analysis of what we know," Emerging Markets Review, Elsevier, vol. 55(C).
    18. Arte, Pratik & Larimo, Jorma, 2022. "Moderating influence of product diversification on the international diversification-performance relationship: A meta-analysis," Journal of Business Research, Elsevier, vol. 139(C), pages 1408-1423.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tatiana Kostova & Sjoerd Beugelsdijk, 2021. "Integrating Diversity into Distance Research for Added Rigor, Parsimony, and Relevance," Journal of Management Studies, Wiley Blackwell, vol. 58(6), pages 1669-1689, September.
    2. Miriam Flickinger & Miriam Zschoche, 2023. "The Interplay of Subsidiary Performance and Cultural Distance in International Downsizing Decisions," Schmalenbach Journal of Business Research, Springer, vol. 75(2), pages 127-147, June.
    3. Sjoerd Beugelsdijk, 2022. "Capitalizing on the uniqueness of international business: Towards a theory of place, space, and organization," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(9), pages 2050-2067, December.
    4. Christopher Hansen & Holger Steinmetz & Jörn Block, 2022. "How to conduct a meta-analysis in eight steps: a practical guide," Management Review Quarterly, Springer, vol. 72(1), pages 1-19, February.
    5. Tatiana Kostova & Sjoerd Beugelsdijk & W. Richard Scott & Vincent E. Kunst & Chei Hwee Chua & Marc Essen, 2020. "The construct of institutional distance through the lens of different institutional perspectives: Review, analysis, and recommendations," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(4), pages 467-497, June.
    6. Moalla, Emna & Mayrhofer, Ulrike, 2020. "How does distance affect market entry mode choice? Evidence from French companies," European Management Journal, Elsevier, vol. 38(1), pages 135-145.
    7. Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    8. Zhang, Wei & He, Xinming & Wang, Tao & Wang, Kang, 2023. "Institutional distance and the international market entry mode: A meta-analysis," Journal of International Management, Elsevier, vol. 29(1).
    9. Schmid, David & Morschett, Dirk, 2020. "Decades of research on foreign subsidiary divestment: What do we really know about its antecedents?," International Business Review, Elsevier, vol. 29(4).
    10. Sasa Ding & Frank McDonald & Yingqi Wei, 2021. "Is Internationalization Beneficial to Innovation? Evidence from a Meta-analysis," Management International Review, Springer, vol. 61(4), pages 469-519, August.
    11. Jeoung Yul Lee & Vasyl Taras & Alfredo Jiménez & Byungchul Choi & Chinmay Pattnaik, 2020. "Ambidextrous Knowledge Sharing within R&D Teams and Multinational Enterprise Performance: The Moderating Effects of Cultural Distance in Uncertainty Avoidance," Management International Review, Springer, vol. 60(3), pages 387-425, June.
    12. Oded Shenkar & Stephen B Tallman & Hao Wang & Jie Wu, 2022. "National culture and international business: A path forward," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(3), pages 516-533, April.
    13. Francesca Talamini & Gianmarco Altoè & Barbara Carretti & Massimo Grassi, 2017. "Musicians have better memory than nonmusicians: A meta-analysis," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-21, October.
    14. Palitha Konara & Alexander Mohr, 2019. "Why We Should Stop Using the Kogut and Singh Index," Management International Review, Springer, vol. 59(3), pages 335-354, June.
    15. Rosalie L Tung & Günter K Stahl, 2018. "The tortuous evolution of the role of culture in IB research: What we know, what we don’t know, and where we are headed," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1167-1189, December.
    16. Obsa Urgessa Ayana & Jima Degaga, 2022. "Effects of rural electrification on household welfare: a meta-regression analysis," International Review of Economics, Springer;Happiness Economics and Interpersonal Relations (HEIRS), vol. 69(2), pages 209-261, June.
    17. Josef C. Brada & Ichiro Iwasaki, 2022. "The Effect of Target-Country Institutions on Cross-Border Merger and Acquisition Activity: A Quantitative Literature Survey," Econometric Research in Finance, SGH Warsaw School of Economics, Collegium of Economic Analysis, vol. 7(1), pages 1-70.
    18. Simon Hartmann & Thomas Lindner & Jakob Müllner & Jonas Puck, 2022. "Beyond the nation-state: Anchoring supranational institutions in international business research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 53(6), pages 1282-1306, August.
    19. Mattia Filomena & Matteo Picchio, 2023. "Retirement and health outcomes in a meta‐analytical framework," Journal of Economic Surveys, Wiley Blackwell, vol. 37(4), pages 1120-1155, September.
    20. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:jintbs:v:52:y:2021:i:1:d:10.1057_s41267-020-00385-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.palgrave-journals.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.