IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i3p495-514.html

How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations

Author

Listed:
  • Burt S. Barnow
  • Sanjay K. Pandey
  • Qian “Eric†Luo

Abstract

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

Suggested Citation

  • Burt S. Barnow & Sanjay K. Pandey & Qian “Eric†Luo, 2024. "How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations," Evaluation Review, , vol. 48(3), pages 495-514, June.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:495-514
    DOI: 10.1177/0193841X241227480
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241227480
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241227480?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Chelsea Richwine & Qian Eric Luo & Zoë Thorkildsen & Nicholas J. Chong & Rebecca Morris & Burt S. Barnow & Sanjay K. Pandey, 2022. "Defining And Assessing The Value Of Canonical Mixed Methods Research Designs In Public Policy And Public Administration," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 41(3), pages 891-920, June.
    3. de Milliano, Marlous & Barrington, Clare & Angeles, Gustavo & Gbedemah, Christiana, 2021. "Crowding-out or crowding-in? Effects of LEAP 1000 unconditional cash transfer program on household and community support among women in rural Ghana," World Development, Elsevier, vol. 143(C).
    4. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, 2016. "Impact Evaluation in Practice, Second Edition," World Bank Publications - Books, The World Bank Group, number 25030, April.
    5. Rao, Vijayendra & Ananthpur, Kripa & Malik, Kabir, 2017. "The Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India," World Development, Elsevier, vol. 99(C), pages 481-497.
    6. Barnow, Burt S., 2010. "Setting up social experiments: the good, the bad, and the ugly," Zeitschrift für ArbeitsmarktForschung - Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 43(2), pages 91-105.
    7. Martha A. Starr, 2014. "Qualitative And Mixed-Methods Research In Economics: Surprising Growth, Promising Future," Journal of Economic Surveys, Wiley Blackwell, vol. 28(2), pages 238-264, April.
    8. David Greenberg & Karl Ashworth & Andreas Cebulla & Robert Walker, 2005. "When Welfare-to-Work Programs Seem to Work Well: Explaining Why Riverside and Portland Shine So Brightly," ILR Review, Cornell University, ILR School, vol. 59(1), pages 34-50, October.
    9. Roelen, Keetie & Saha, Amrita, 2021. "Pathways to stronger futures? The role of social protection in reducing psychological risk factors for child development in Haiti," World Development, Elsevier, vol. 142(C).
    10. Imbens, Guido W., 2018. "Comments On: Understanding and Misunderstanding Randomized Controlled Trails by Cartwright and Deaton," Research Papers 3648, Stanford University, Graduate School of Business.
    11. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Quan Nha Hong & Sergi FÃ bregues, 2026. "A Critical Reflection of Generalization in Mixed Methods Research," Evaluation Review, , vol. 50(2), pages 200-230, April.
    2. Anne Revillard, 2026. "External Validity and Generalizability in Program Evaluation: Embracing Complexity," Evaluation Review, , vol. 50(2), pages 183-199, April.
    3. Thomas Delahais, 2026. "External Validity for Complex Interventions: Towards Pragmatic Strategies for Evaluators," Evaluation Review, , vol. 50(2), pages 256-278, April.
    4. Thilo Bodenstein & Achim Kemmerling, 2026. "Taking Stock of Qualitative Methods of Evaluation: A Study of Practices and Quality Criteria," Evaluation Review, , vol. 50(1), pages 89-115, February.
    5. Rabia Majeed & Javed Iqbal & Zahoor ul Haq & Muhammad Faisal Shahzad, 2025. "Empowering Expecting Women to Obtain High-Quality Healthcare in Pakistan: An Evaluation of Khyber Pakhtunkhwa’s Conditional Cash Transfer Program," Evaluation Review, , vol. 49(2), pages 209-236, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    2. Sandeep Devanatha Pillai & Brent Goldfarb & David Kirsch, 2024. "Lovely and likely: Using historical methods to improve inference to the best explanation in strategy," Strategic Management Journal, Wiley Blackwell, vol. 45(8), pages 1539-1566, August.
    3. Jean Cartier-Bresson, 2013. "Le pouvoir du positivisme et ses limites : microéconométrie et macroéconométrie actuelles du développement," Working Papers hal-00847005, HAL.
    4. Cyrus J. DiCiccio & Joseph P. Romano & Michael Wolf, 2016. "Improving weighted least squares inference," ECON - Working Papers 232, Department of Economics - University of Zurich, revised Nov 2017.
    5. Romano, Joseph P. & Wolf, Michael, 2017. "Resurrecting weighted least squares," Journal of Econometrics, Elsevier, vol. 197(1), pages 1-19.
    6. Carlianne Patrick & Amanda Ross & Heather Stephens, 2016. "Designing Policies to Spur Economic Growth: How Regional Scientists Can Contribute to Future Policy Development and Evaluation," Working Papers 16-04, Department of Economics, West Virginia University.
    7. P. Dorian Owen, 2017. "Evaluating Ingenious Instruments for Fundamental Determinants of Long-Run Economic Growth and Development," Econometrics, MDPI, vol. 5(3), pages 1-33, September.
    8. Davidson Heath & Matthew C. Ringgenberg & Mehrdad Samadi & Ingrid M. Werner, 2023. "Reusing Natural Experiments," Journal of Finance, American Finance Association, vol. 78(4), pages 2329-2364, August.
    9. van Dijk, M.A., 2014. "The Social Value of Finance," ERIM Inaugural Address Series Research in Management EIA-2014-055-F&A, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam..
    10. Eric Monnet & Francois R. Velde, 2020. "Money, Banking, and Old-School Historical Economics," Working Paper Series WP-2020-28, Federal Reserve Bank of Chicago.
    11. Gassen, Joachim, 2014. "Causal inference in empirical archival financial accounting research," Accounting, Organizations and Society, Elsevier, vol. 39(7), pages 535-544.
    12. Boris Salazar-Trujillo & Daniel Otero Robles, 2019. "La revolución empírica en economía," Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 38(68), pages 15-48.
    13. Raquel Bernal & Michele Giannola & Milagros Nores, 2022. "The Effect of Center-Based Early Education on Disadvantaged Children's Developmental Trajectories: Experimental Evidence from Colombia," Working Papers 2022-027, Human Capital and Economic Opportunity Working Group.
    14. Rao, Vijayendra, 2020. "Evidence-based development needs a diversity of tools, with a bottom-up process of “embedded” dialogue," World Development, Elsevier, vol. 127(C).
    15. James J. Heckman, 2010. "Building Bridges between Structural and Program Evaluation Approaches to Evaluating Policy," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 356-398, June.
    16. Niu, Yongzhi, 2010. "Taxpayers' Response to Warnings of a Possible Tax Audit: Do They Change Their Compliance Behavior?," MPRA Paper 25551, University Library of Munich, Germany.
    17. Andreas A. Andrikopoulos & Dimitrios C. Gkountanis, 2011. "Issues and Models in Applied Econometrics: A partial survey," South-Eastern Europe Journal of Economics, Association of Economic Universities of South and Eastern Europe and the Black Sea Region, vol. 9(2), pages 107-165.
    18. Scott A. Carson, 2010. "A Quantile Approach to the Relationship between Body Mass, Wealth, and Inequality," CESifo Working Paper Series 3288, CESifo.
    19. Nikolas Kuschnig & Gregor Zens & Jesús Crespo Cuaresma, 2021. "Hidden in Plain Sight: Influential Sets in Linear Models," CESifo Working Paper Series 8981, CESifo.
    20. Robert A. Lawson & Ryan Murphy & Benjamin Powell, 2020. "The Determinants Of Economic Freedom: A Survey," Contemporary Economic Policy, Western Economic Association International, vol. 38(4), pages 622-642, October.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:495-514. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.