IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i3p427-460.html
   My bibliography  Save this article

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Author

Listed:
  • Julia H. Littell

Abstract

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of “evidence-based†psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Suggested Citation

  • Julia H. Littell, 2024. "The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations," Evaluation Review, , vol. 48(3), pages 427-460, June.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:427-460
    DOI: 10.1177/0193841X241227481
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241227481
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241227481?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Charles F. Manski, 2013. "Response to the Review of ‘Public Policy in an Uncertain World’," Economic Journal, Royal Economic Society, vol. 0, pages 412-415, August.
    2. Kerry Dwan & Douglas G Altman & Mike Clarke & Carrol Gamble & Julian P T Higgins & Jonathan A C Sterne & Paula R Williamson & Jamie J Kirkham, 2014. "Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials," PLOS Medicine, Public Library of Science, vol. 11(6), pages 1-22, June.
    3. Issa J. Dahabreh & Sarah E. Robertson & Lucia C. Petito & Miguel A. Hernán & Jon A. Steingrimsson, 2023. "Efficient and robust methods for causally interpretable meta‐analysis: Transporting inferences from multiple randomized trials to a target population," Biometrics, The International Biometric Society, vol. 79(2), pages 1057-1072, June.
    4. Williams, Martin J., 2019. "External validity and policy adaptation: From impact evaluation to policy design," PEGNet Policy Briefs 18/2019, PEGNet - Poverty Reduction, Equity and Growth Network, Kiel Institute for the World Economy (IfW Kiel).
    5. Manski, Charles F., 2013. "Public Policy in an Uncertain World: Analysis and Decisions," Economics Books, Harvard University Press, number 9780674066892, Spring.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Muller, Seán M., 2021. "The dangers of performative scientism as the alternative to anti-scientific policymaking: A critical, preliminary assessment of South Africa’s Covid-19 response and its consequences," World Development, Elsevier, vol. 140(C).
    2. Raffaella Giacomini & Toru Kitagawa, 2021. "Robust Bayesian Inference for Set‐Identified Models," Econometrica, Econometric Society, vol. 89(4), pages 1519-1556, July.
    3. Wolfgang Frimmel & Martin Halla & Rudolf Winter-Ebmer, 2016. "How Does Parental Divorce Affect Children's Long-term Outcomes?," Working Papers 2016-13, Faculty of Economics and Statistics, Universität Innsbruck.
    4. Eric Danan & Thibault Gajdos & Brian Hill & Jean-Marc Tallon, 2016. "Robust Social Decisions," American Economic Review, American Economic Association, vol. 106(9), pages 2407-2425, September.
    5. Fernando Hoces de la Guardia & Sean Grant & Edward Miguel, 2021. "A framework for open policy analysis," Science and Public Policy, Oxford University Press, vol. 48(2), pages 154-163.
    6. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    7. Xiaoyu Cheng, 2022. "Robust Data-Driven Decisions Under Model Uncertainty," Papers 2205.04573, arXiv.org.
    8. Hill, Brian, 2023. "Beyond uncertainty aversion," Games and Economic Behavior, Elsevier, vol. 141(C), pages 196-222.
    9. Thomas R. Harris, 2017. "Incorporating Risk in Analysis of Tax Policies for Solar Power Investments," International Journal of Energy Economics and Policy, Econjournals, vol. 7(6), pages 112-118.
    10. Choudhury, Sanchari, 2019. "WTO membership and corruption," European Journal of Political Economy, Elsevier, vol. 60(C).
    11. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    12. T. N. Srinivasan, 2014. "Development Process: A Manual or a Collection of Anecdotes?," Journal of Human Development and Capabilities, Taylor & Francis Journals, vol. 15(4), pages 418-423, November.
    13. Anthony Perl & Michael Howlett & M. Ramesh, 2018. "Policy-making and truthiness: Can existing policy models cope with politicized evidence and willful ignorance in a “post-fact” world?," Policy Sciences, Springer;Society of Policy Sciences, vol. 51(4), pages 581-600, December.
    14. Luc Behaghel & Karen Macours & Julie Subervie, 2019. "How can randomised controlled trials help improve the design of the common agricultural policy?," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 46(3), pages 473-493.
    15. Ashesh Rambachan & Jonathan Roth, 2023. "A More Credible Approach to Parallel Trends," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2555-2591.
    16. Ian W. R. Martin & Robert S. Pindyck, 2015. "Averting Catastrophes: The Strange Economics of Scylla and Charybdis," American Economic Review, American Economic Association, vol. 105(10), pages 2947-2985, October.
    17. Takanori Ida & Takunori Ishihara & Koichiro Ito & Daido Kido & Toru Kitagawa & Shosei Sakaguchi & Shusaku Sasaki, 2021. "Paternalism, Autonomy, or Both? Experimental Evidence from Energy Saving Programs," Papers 2112.09850, arXiv.org.
    18. Claudia M. Buch & Oliver Holtemöller, 2014. "Do we need new modelling approaches in macroeconomics?," Chapters, in: Ewald Nowotny & Doris Ritzberger-Grünwald & Peter Backé (ed.), Financial Cycles and the Real Economy, chapter 3, pages 36-58, Edward Elgar Publishing.
    19. Alfredo Di Tillio & Marco Ottaviani & Peter Norman Sørensen, 2017. "Persuasion Bias in Science: Can Economics Help?," Economic Journal, Royal Economic Society, vol. 127(605), pages 266-304, October.
    20. Ken Binmore, 2016. "A minimal extension of Bayesian decision theory," Theory and Decision, Springer, vol. 80(3), pages 341-362, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:427-460. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.