IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v36y2012i5p375-401.html
   My bibliography  Save this article

When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model

Author

Listed:
  • Diana Epstein
  • Jacob Alex Klerman

Abstract

Background: Recent reviews suggest that many plausible programs are found to have at best small impacts not commensurate with their cost, and often have no detectable positive impacts at all. Even programs with initial rigorous impact evaluation (RIE) that show them to be effective often fail a second test with an expanded population or at multiple sites. Objective: This article argues that more rapid movement to RIE is a partial cause of the low success rate of RIE and proposes a constructive response: process evaluations that compare program intermediate outcomes—in the treatment group, during the operation of the program—against a more falsifiable extension of the conventional logic model. Conclusion: Our examples suggest that such process evaluations would allow funders to deem many programs unlikely to show impacts and therefore not ready for random assignment evaluation—without the high cost and long time lines of an RIE. The article then develops the broader implications of such a process analysis step for broader evaluation strategy.

Suggested Citation

  • Diana Epstein & Jacob Alex Klerman, 2012. "When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model," Evaluation Review, , vol. 36(5), pages 375-401, October.
    • Handle: RePEc:sae:evarev:v:36:y:2012:i:5:p:375-401
    DOI: 10.1177/0193841X12474275
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X12474275
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X12474275?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:mpr:mprres:4971 is not listed on IDEAS
    2. McLaughlin, John A. & Jordan, Gretchen B., 1999. "Logic models: a tool for telling your programs performance story," Evaluation and Program Planning, Elsevier, vol. 22(1), pages 65-72.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gates, Lauren B. & Pearlmutter, Sue & Keenan, Kat & Divver, Caitlin & Gorroochurn, Prakash, 2018. "Career readiness programming for youth in foster care," Children and Youth Services Review, Elsevier, vol. 89(C), pages 152-164.
    2. Ofek, Yuval, 2017. "Evaluating social exclusion interventions in university-community partnerships," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 46-55.
    3. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    4. Thierry Bastelaer & Benjamin Woodman & Minki Chatterji & David Long, 2016. "Saving for delivery through technology: an inquiry into an electronic commitment savings product for maternal healthcare in Kenya," International Journal of Health Planning and Management, Wiley Blackwell, vol. 31(4), pages 254-272, October.
    5. Elisa Ricciuti & Francesca Calò, 2018. "Are foundations assessing their impact? Concepts, methods and barriers to social impact assessment in Italian foundations," International Review on Public and Nonprofit Marketing, Springer;International Association of Public and Non-Profit Marketing, vol. 15(4), pages 553-574, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tim Benijts, 2014. "A Business Sustainability Model for Government Corporations. A Belgian Case Study," Business Strategy and the Environment, Wiley Blackwell, vol. 23(3), pages 204-216, March.
    2. Fielden, Sarah J. & Rusch, Melanie L. & Masinda, Mambo Tabu & Sands, Jim & Frankish, Jim & Evoy, Brian, 2007. "Key considerations for logic model development in research partnerships: A Canadian case study," Evaluation and Program Planning, Elsevier, vol. 30(2), pages 115-124, May.
    3. Ebenso, Bassey & Manzano, Ana & Uzochukwu, Benjamin & Etiaba, Enyi & Huss, Reinhard & Ensor, Tim & Newell, James & Onwujekwe, Obinna & Ezumah, Nkoli & Hicks, Joe & Mirzoev, Tolib, 2019. "Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 97-110.
    4. Wasserman, Deborah L., 2010. "Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations," Evaluation and Program Planning, Elsevier, vol. 33(2), pages 67-80, May.
    5. Voeten, J.J., 2012. "Understanding responsible innovation in small producers’ clusters in Northern Vietnam : A grounded theory approach to globalization and poverty alleviation," Other publications TiSEM e01da02b-ef2b-47c9-8d06-4, Tilburg University, School of Economics and Management.
    6. Peyton, David J. & Scicchitano, Michael, 2017. "Devil is in the details: Using logic models to investigate program process," Evaluation and Program Planning, Elsevier, vol. 65(C), pages 156-162.
    7. Matt Andrews, 2022. "This is How to Think About and Achieve Public Policy Success," CID Working Papers 413, Center for International Development at Harvard University.
    8. Wifo, 2021. "WIFO-Monatsberichte, Heft 10/2021," WIFO Monatsberichte (monthly reports), WIFO, vol. 94(10), October.
    9. Sobelson, Robyn K. & Young, Andrea C., 2013. "Evaluation of a federally funded workforce development program: The Centers for Public Health Preparedness," Evaluation and Program Planning, Elsevier, vol. 37(C), pages 50-57.
    10. Wu, Huang & Shen, Jianping & Jones, Jeffrey & Gao, Xingyuan & Zheng, Yunzheng & Krenn, Huilan Y., 2019. "Using logic model and visualization to conduct portfolio evaluation," Evaluation and Program Planning, Elsevier, vol. 74(C), pages 69-75.
    11. O'Keefe, Christine M. & Head, Richard J., 2011. "Application of logic models in a large scientific research program," Evaluation and Program Planning, Elsevier, vol. 34(3), pages 174-184, August.
    12. Vinícius P. Rodrigues & Daniela C. A. Pigosso & Jakob W. Andersen & Tim C. McAloone, 2018. "Evaluating the Potential Business Benefits of Ecodesign Implementation: A Logic Model Approach," Sustainability, MDPI, vol. 10(6), pages 1-26, June.
    13. Chanel Bjanca V. Balinbin & Krystina Trizia R. Balatbat & Alyssa Nicolette B. Balayan & Maria Isabel C. Balcueva & Mary Grace B. Balicat & Thea Arabelle S. Balidoy & John Rey B. Macindo & Gian Carlo S, 2020. "Occupational determinants of compassion satisfaction and compassion fatigue among Filipino registered nurses," Journal of Clinical Nursing, John Wiley & Sons, vol. 29(5-6), pages 955-963, March.
    14. Janger, Jürgen & Schubert, Torben & Andries, Petra & Rammer, Christian & Hoskens, Machteld, 2017. "The EU 2020 innovation indicator: A step forward in measuring innovation outputs and outcomes?," Research Policy, Elsevier, vol. 46(1), pages 30-42.
    15. Jürgen Janger & Agnes Kügler, 2018. "Innovationseffizienz. Österreich im internationalen Vergleich," WIFO Studies, WIFO, number 61111, April.
    16. Fangyuan Chang & Andrea Eriksson & Britt Östlund, 2020. "Discrepancies between Expected and Actual Implementation: The Process Evaluation of PERS Integration in Nursing Homes," IJERPH, MDPI, vol. 17(12), pages 1-18, June.
    17. Louise R. Manfredi & Meriel Stokoe & Rebecca Kelly & Seyeon Lee, 2021. "Teaching Sustainable Responsibility through Informal Undergraduate Design Education," Sustainability, MDPI, vol. 13(15), pages 1-13, July.
    18. Daniel Nigohosyan & Albena Vutsova, 2018. "The 2014–2020 European Regional Development Fund Indicators: The Incomplete Evolution," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 137(2), pages 559-577, June.
    19. Laura Kreiling & Ahmed Bounfour, 2020. "A practice-based maturity model for holistic TTO performance management: development and initial use," The Journal of Technology Transfer, Springer, vol. 45(6), pages 1718-1747, December.
    20. Voeten, J. & Naudé, Wim, 2013. "Internal Regulation of Innovation Externalities for Development : Lessons from Vietnam," Other publications TiSEM fcbdcd1e-ccf9-4b33-952d-8, Tilburg University, School of Economics and Management.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:36:y:2012:i:5:p:375-401. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.