IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v46y2022i5p469-516.html

Using Iterative Experimentation to Accelerate Program Improvement: A Case Example

Author

Listed:
  • Rebecca A. Maynard
  • Rebecca N. Baelen
  • David Fein
  • Phomdaen Souvanna

Abstract

Background: This article offers a case example of how experimental evaluation methods can be coupled with principles of design-based implementation research (DBIR), improvement science (IS), and rapid-cycle evaluation (RCE) methods to provide relatively quick, low-cost, credible assessments of strategies designed to improve programs, policies, or practices. Objectives: This article demonstrates the feasibility and benefits of blending DBIR, IS, and RCE practices with embedded randomized controlled trials (RCTs) to improve the pace and efficiency of program improvement. Research design: This article describes a two-cycle experimental test of staff-designed strategies for improving a workforce development program. Youth enrolled in Year Up’s Professional Training Corps (PTC) programs were randomly assigned to “improvement strategies†designed to boost academic success and persistence through the 6-month learning and development (L&D) phase of the program, when participants spend most of their program-related time in courses offered by partner colleges. Subjects: The study sample includes 317 youth from three PTC program sites. Measures: The primary outcome measures are completion of the program’s L&D phase and continued college enrollment beyond the L&D phase. Results: The improvement strategies designed and tested during the study increased program retention through L&D by nearly 10 percentage points and increased college persistence following L&D by 13 percentage points. Conclusion: Blending DBIR, IS, and RCE principles with a multi-cycle RCT generated highly credible estimates of the efficacy of the tested improvement strategies within a relatively short period of time (18 months) at modest cost and with reportedly low burden for program staff.

Suggested Citation

  • Rebecca A. Maynard & Rebecca N. Baelen & David Fein & Phomdaen Souvanna, 2022. "Using Iterative Experimentation to Accelerate Program Improvement: A Case Example," Evaluation Review, , vol. 46(5), pages 469-516, October.
    • Handle: RePEc:sae:evarev:v:46:y:2022:i:5:p:469-516
    DOI: 10.1177/0193841X20923199
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20923199
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20923199?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:mpr:mprres:8169 is not listed on IDEAS
    2. repec:mpr:mprres:8164 is not listed on IDEAS
    3. Kane, Thomas J & Rouse, Cecilia Elena, 1995. "Labor-Market Returns to Two- and Four-Year College," American Economic Review, American Economic Association, vol. 85(3), pages 600-614, June.
    4. Burtless, Gary & Hausman, Jerry A, 1978. "The Effect of Taxation on Labor Supply: Evaluating the Gary Negative Income Tax Experiments," Journal of Political Economy, University of Chicago Press, vol. 86(6), pages 1103-1130, December.
    5. Dave E. Marcotte, 2010. "The Earnings Effect Of Education At Community Colleges," Contemporary Economic Policy, Western Economic Association International, vol. 28(1), pages 36-51, January.
    6. repec:mpr:mprres:2737 is not listed on IDEAS
    7. repec:mpr:mprres:6649 is not listed on IDEAS
    8. Louis Jacobson & Robert J. Lalonde & Daniel Sullivan, 2005. "The Impact of Community College Retraining on Older Displaced Workers: Should We Teach Old Dogs New Tricks?," ILR Review, Cornell University, ILR School, vol. 58(3), pages 398-415, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Celeste K. Carruthers & Christopher Jepsen, 2020. "Vocational Education: An International Perspective," CESifo Working Paper Series 8718, CESifo.
    2. Stenberg, Anders & Westerlund, Olle, 2016. "Flexibility at a cost – Should governments stimulate tertiary education for adults?," The Journal of the Economics of Ageing, Elsevier, vol. 7(C), pages 69-86.
    3. Blomquist, Glenn C. & Coomes, Paul A. & Jepsen, Christopher & Koford, Brandon C. & Troske, Kenneth R., 2014. "Estimating the social value of higher education: willingness to pay for community and technical colleges," Journal of Benefit-Cost Analysis, Cambridge University Press, vol. 5(1), pages 3-41, January.
    4. Aucejo, Esteban M. & Hupkau, Claudia & Ruiz-Valenzuela, Jenifer, 2025. "Where versus what: college value-added and returns to field of study in further education," LSE Research Online Documents on Economics 118067, London School of Economics and Political Science, LSE Library.
    5. Dave E. Marcotte, 2019. "The Returns to Education at Community Colleges: New Evidence from the Education Longitudinal Survey," Education Finance and Policy, MIT Press, vol. 14(4), pages 523-547, Fall.
    6. Chung, Anna, 2008. "The Effects of For-Profit College Training on Earnings," MPRA Paper 18972, University Library of Munich, Germany, revised May 2009.
    7. Celeste K. Carruthers & Thomas Sanford, 2015. "Way Station or Launching Pad? Unpacking the Returns to Postsecondary Adult Education," Working Papers 2015-02, University of Tennessee, Department of Economics.
    8. Stenberg, Anders, 2022. "Does formal education for adults yield long-term multiplier effects or human capital depreciation?," Economics of Education Review, Elsevier, vol. 90(C).
    9. Lockwood Reynolds, C., 2012. "Where to attend? Estimating the effects of beginning college at a two-year institution," Economics of Education Review, Elsevier, vol. 31(4), pages 345-362.
    10. Schanzenbach, Diane W. & Turner, Sarah, 2022. "Limited supply and lagging enrollment: Production technologies and enrollment changes at community colleges during the pandemic," Journal of Public Economics, Elsevier, vol. 212(C).
    11. Ann Huff Stevens & Michal Kurlaender & Michel Grosz, 2019. "Career Technical Education and Labor Market Outcomes: Evidence from California Community Colleges," Journal of Human Resources, University of Wisconsin Press, vol. 54(4), pages 986-1036.
    12. Jepsen, Christopher & Montgomery, Mark, 2009. "Miles to go before I learn: The effect of travel distance on the mature person's choice of a community college," Journal of Urban Economics, Elsevier, vol. 65(1), pages 64-73, January.
    13. Carruthers, Celeste K. & Sanford, Thomas, 2018. "Way station or launching pad? Unpacking the returns to adult technical education," Journal of Public Economics, Elsevier, vol. 165(C), pages 146-159.
    14. Stenberg, Anders, 2022. "A Note on Evaluating Formal Education for Adults," IZA Discussion Papers 15379, IZA Network @ LISER.
    15. Gan, Li & Ju, Gaosheng & Zhu, Xi, 2015. "Nonparametric estimation of structural labor supply and exact welfare change under nonconvex piecewise-linear budget sets," Journal of Econometrics, Elsevier, vol. 188(2), pages 526-544.
    16. Olmstead, Sheila M. & Michael Hanemann, W. & Stavins, Robert N., 2007. "Water demand under alternative price structures," Journal of Environmental Economics and Management, Elsevier, vol. 54(2), pages 181-198, September.
    17. Isaak, Niklas & Jessen, Robin, 2024. "Normative judgments implicit in the tax system: A simulation approach," Ruhr Economic Papers 1128, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    18. Wayne A. Grove & Andrew Hussey, 2011. "Returns To Field Of Study Versus School Quality: Mba Selection On Observed And Unobserved Heterogeneity," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 730-749, July.
    19. Matthew Higgins & Daniel Levy & Andrew T. Young, 2003. "Growth and Convergence across the US: Evidence from County-Level Data," Working Papers 2003-03, Bar-Ilan University, Department of Economics.
    20. Alena Bicakova & Guido Matias Cortes & Kelly Foley & Jacopo Mazza & Peter McHenry, 2025. "Unpacking the Countercyclicality of Post-Secondary Enrollment in the United States," CERGE-EI Working Papers wp795, The Center for Economic Research and Graduate Education - Economics Institute, Prague.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:46:y:2022:i:5:p:469-516. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.