IDEAS home Printed from https://ideas.repec.org/p/cid/wpfacu/428.html
   My bibliography  Save this paper

Adaptive Evaluation: A Complexity-Based Approach to Systematic Learning for Innovation and Scaling in Development

Author

Listed:
  • Michael Walton

Abstract

Nearly all challenges in international development tend to be complex because they depend on constantly evolving human behaviour, systems, and contexts, involving multiple actors, entities, and processes. As a result, both the discovery and scaling of innovations to address challenges in development often involve changes in system behaviour or even system-level transformation. This is rarely a linear process over time and can result in unexpected outcomes. Existing evaluation techniques commonly used in international development, including Randomized Control Trials (RCT) and quasi-experimental methods, are good at assessing specific effects of interventions but are not designed for the change processes inherent to innovation and scaling within a system. There is a need to reconstruct how we use existing measurement tools, techniques, and methodologies so that they capture the complexity of the environment in which an intervention or change occurs. We introduce Adaptive Evaluation, designed to learn at various levels of complexity while supporting the transformation needed to foster sustainable change. An Adaptive Evaluation uses three main approaches to work with complex questions—systems diagnosis, theory-based assessment of change processes, and iterative designs. An Adaptive Evaluation typically builds hypotheses from field-based interactions, emphasizes learning over testing, advocates open-mindedness with techniques, and appreciates the value of dialogue and participation in navigating complex processes. It can use RCT or similar techniques to analyse specific processes within a system or a development cycle, but these are embedded in a broader approach to assessment and interpretation. It is designed to be flexible and adjust to shifting contexts. Finally, an Adaptive Evaluation can be applied at any stage in a complex intervention's lifecycle, from the interpretation of the system and change processes to rapid experimentation, prototyping, and testing of select interventions, and then adaptation to different settings for impact at scale. This paper provides the theoretical basis for an Adaptive Evaluation—the main approaches, core ideology, process, and applications.

Suggested Citation

  • Michael Walton, 2023. "Adaptive Evaluation: A Complexity-Based Approach to Systematic Learning for Innovation and Scaling in Development," CID Working Papers 428, Center for International Development at Harvard University.
  • Handle: RePEc:cid:wpfacu:428
    as

    Download full text from publisher

    File URL: https://www.hks.harvard.edu/sites/default/files/centers/cid/files/publications/faculty-working-papers/428_Gokhale_Walton.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    3. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukherji & Marc Shotland & Michael Walton, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India," NBER Working Papers 22746, National Bureau of Economic Research, Inc.
    4. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    5. Andrews, Matt & Pritchett, Lant & Woolcock, Michael, 2017. "Building State Capability: Evidence, Analysis, Action," OUP Catalogue, Oxford University Press, number 9780198747482.
    6. Jesus Felipe & Norio Usui & Arnelyn Abdon, 2011. "Rethinking The Growth Diagnostics Approach: Questions From The Practitioners," Journal of International Commerce, Economics and Policy (JICEP), World Scientific Publishing Co. Pte. Ltd., vol. 2(02), pages 251-276.
    7. Abhijit V. Banerjee & Rukmini Banerji & Esther Duflo & Rachel Glennerster & Stuti Khemani, 2010. "Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India," American Economic Journal: Economic Policy, American Economic Association, vol. 2(1), pages 1-30, February.
    8. Gunnar Myrdal, 1974. "What Is Development?," Journal of Economic Issues, Taylor & Francis Journals, vol. 8(4), pages 729-736, December.
    9. Sudha Narayanan & Christian Oldiges & Shree Saha, 2020. "Employment guarantee during times of COVID-19: Pro-poor and pro-return-migrant?," Indira Gandhi Institute of Development Research, Mumbai Working Papers 2020-034, Indira Gandhi Institute of Development Research, Mumbai, India.
    10. Abhijit Banerjee & Rukmini Banerji & Esther Duflo, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level†in India," Working Papers id:11419, eSocialSciences.
    11. Zidong An & João Tovar Jalles & Prakash Loungani, 2018. "How well do economists forecast recessions?," International Finance, Wiley Blackwell, vol. 21(2), pages 100-121, June.
    12. Maximilian Kasy & Anja Sautmann, 2021. "Adaptive Treatment Assignment in Experiments for Policy Choice," Econometrica, Econometric Society, vol. 89(1), pages 113-132, January.
    13. Walton, Mat, 2014. "Applying complexity theory: A review to inform evaluation design," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 119-126.
    14. ., 2022. "Impact of policies toward overcrowding: case studies," Chapters, in: Public Sector Entrepreneurship, chapter 7, pages 62-96, Edward Elgar Publishing.
    15. Yong Tao, 2016. "Spontaneous economic order," Journal of Evolutionary Economics, Springer, vol. 26(3), pages 467-500, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    2. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    3. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    4. Annie Duflo & Jessica Kiessel & Adrienne Lucas, 2020. "Experimental Evidence on Alternative Policies to Increase Learning at Scale," NBER Working Papers 27298, National Bureau of Economic Research, Inc.
    5. Victor Lavy & Assaf Kott & Genia Rachkovski, 2022. "Does Remedial Education in Late Childhood Pay Off After All? Long-Run Consequences for University Schooling, Labor Market Outcomes, and Intergenerational Mobility," Journal of Labor Economics, University of Chicago Press, vol. 40(1), pages 239-282.
    6. Emily Beam & Priya Mukherjee & Laia Navarro-Sola, 2022. "Lowering Barriers to Remote Education: Experimental Impacts on Parental Responses and Learning," Working Papers 2022-030, Human Capital and Economic Opportunity Working Group.
    7. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    8. Perera, Marcelo & Aboal, Diego, 2019. "The impact of a mathematics computer-assisted learning platform on students' mathematics test scores," MERIT Working Papers 2019-007, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    9. Yasuyuki Sawada & Minhaj Mahmud & Mai Seki & An Le & Hikaru Kawarazaki, 2019. "Fighting the Learning Crisis in Developing Countries: A Randomized Experiment of Self-Learning at the Right Level," CIRJE F-Series CIRJE-F-1127, CIRJE, Faculty of Economics, University of Tokyo.
    10. Francesca Marchetta & Tom Dilly, 2019. "Supporting Education in Africa: Opportunities and Challenges for an Impact Investor," Working Papers hal-02288103, HAL.
    11. Burchardi, Konrad & de Quidt, Jonathan & Gulesci, Selim & Sulaiman, Munshi, 2021. "Credit constraints and demand for remedial education: Evidence from Tanzania," CAGE Online Working Paper Series 545, Competitive Advantage in the Global Economy (CAGE).
    12. Angrist, Noam & de Barros, Andreas & Bhula, Radhika & Chakera, Shiraz & Cummiskey, Chris & DeStefano, Joseph & Floretta, John & Kaffenberger, Michelle & Piper, Benjamin & Stern, Jonathan, 2021. "Building back better to avert a learning catastrophe: Estimating learning loss from COVID-19 school shutdowns in Africa and facilitating short-term and long-term learning recovery," International Journal of Educational Development, Elsevier, vol. 84(C).
    13. Johnston, Jamie & Ksoll, Christopher, 2022. "Effectiveness of interactive satellite-transmitted instruction: Experimental evidence from Ghanaian primary schools," Economics of Education Review, Elsevier, vol. 91(C).
    14. Sawada Yasuyuki & Mahmud Minhaj & Seki Mai & Le An & Kawarazaki Hikaru, 2017. "Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh," Working Papers 156, JICA Research Institute.
    15. Oana Borcan & James Merewood, 2022. "Positive Disruption? Meritocratic Principal Selection and Student Achievement," University of East Anglia School of Economics Working Paper Series 2022-11, School of Economics, University of East Anglia, Norwich, UK..
    16. Li, Haizheng & Liu, Zhiqiang & Yang, Fanzheng & Yu, Li, 2023. "The Impact of Computer-Assisted Instruction on Student Performance: Evidence from the Dual-Teacher Program," IZA Discussion Papers 15944, Institute of Labor Economics (IZA).
    17. de Hoyos, Rafael, 2020. "Equal opportunities to enhance growth," World Development, Elsevier, vol. 127(C).
    18. World Bank, 2018. "World Development Report 2018 [Rapport sur le développement dans le monde 2018]," World Bank Publications - Books, The World Bank Group, number 28340, December.
    19. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukherji & Marc Shotland & Michael Walton, 2016. "Mainstreaming an Effective Intervention: Evidence from Randomized Evaluations of “Teaching at the Right Level” in India," NBER Working Papers 22746, National Bureau of Economic Research, Inc.
    20. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja, 2017. "Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets," American Economic Review, American Economic Association, vol. 107(6), pages 1535-1563, June.

    More about this item

    Keywords

    adaptive evaluation; scaling; innovation; complexity theory; research methodology;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cid:wpfacu:428. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chuck McKenney (email available below). General contact details of provider: https://edirc.repec.org/data/ciharus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.