IDEAS home Printed from https://ideas.repec.org/p/feb/natura/00741.html
   My bibliography  Save this paper

The role of open science practices in scaling evidence-based prevention programs

Author

Listed:
  • Robert Ammerman
  • Anne Duggan
  • John List
  • Lauren Supplee
  • Dana Suskind

Abstract

The goal of creating evidence-based programs is to scale them at sufficient breadth to support population-level improvements in critical outcomes. However, this promise is challenging to fulfill. One of the biggest issues for the field is the reduction in effect sizes seen when a program is taken to scale. This paper discusses an economic perspective that identifies the underlying incentives in the research process that lead to scale up problems and to deliver potential solutions to strengthen outcomes at scale. The principles of open science are well aligned with this goal. One prevention program that has begun to scale across the United States is early childhood home visiting. While there is a substantial impact research on home visiting, overall average effect size is .10 and a recent national randomized trial found attenuated effect sizes in programs implemented under real-world conditions. The paper concludes with a case study of the relevance of the economic model and open science in developing and scaling evidence-based home visiting. The case study considers how the traditional approach for testing interventions has influenced home visiting's evolution to date and how open science practices could have supported efforts to maintain impacts while scaling home visiting. It concludes by considering how open science can accelerate the refinement and scaling of home visiting interventions going forward, through accelerated translation of research into policy and practice.

Suggested Citation

  • Robert Ammerman & Anne Duggan & John List & Lauren Supplee & Dana Suskind, 2021. "The role of open science practices in scaling evidence-based prevention programs," Natural Field Experiments 00741, The Field Experiments Website.
    • Handle: RePEc:feb:natura:00741
    as

    Download full text from publisher

    File URL: http://s3.amazonaws.com/fieldexperiments-papers2/papers/00741.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Charles Michalopoulos & Kristen Faucetta & Carolyn J. Hill & Zimena A. Portilla & Lori Burrell & Helen Lee & Anne Duggan & Virginia Knox, "undated". "Impacts on Family Outcomes of Evidence-Based Early Childhood Home Visiting: Results from the Mother and Infant Home Visiting Program Evaluation," Mathematica Policy Research Reports 3adcbd3368c545679a6784b8a, Mathematica Policy Research.
    2. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    3. Luigi Butera & Philip Grossman & Daniel Houser & John List & Marie-Claire Villeval, 2020. "A New Mechanism to Alleviate the Crises of Confidence in Science - With an Application to the Public Goods Game," Artefactual Field Experiments 00684, The Field Experiments Website.
    4. John A. List, 2020. "Non est Disputandum de Generalizability? A Glimpse into The External Validity Trial," NBER Working Papers 27535, National Bureau of Economic Research, Inc.
    5. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    6. Paulsell, D. & Del Grosso, P. & Supplee, L., 2014. "Supporting replication and scale-up of evidence-based home visiting programs: Assessing the implementation knowledge base," American Journal of Public Health, American Public Health Association, vol. 104(9), pages 1624-1632.
    7. Emily Sama-Miller & Lauren Akers & Andrea Mraz-Esposito & Marykate Zukiewicz & Sarah Avellar & Diane Paulsell & Patricia Del Grosso, "undated". "Home Visiting Evidence of Effectiveness Review: Executive Summary," Mathematica Policy Research Reports 87df04cf10f344939c4ffce2e, Mathematica Policy Research.
    8. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    9. Emily Sama-Miller & Lauren Akers & Andrea Mraz-Esposito & Sarah Avellar & Diane Paulsell & Patricia Del Grosso, "undated". "Home Visiting Evidence of Effectiveness Review (Executive Summary)," Mathematica Policy Research Reports e7ca642d212341c19d252df0b, Mathematica Policy Research.
    10. Omar Al-Ubaydli & John A. List & Dana L. Suskind, 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results," American Economic Review, American Economic Association, vol. 107(5), pages 282-286, May.
    11. Luigi Butera & Philip J Grossman & Daniel Houser & John A List & Marie Claire Villeval, 2020. "A New Mechanism to Alleviate the Crises of Confidence in Science With An Application to the Public Goods GameA Review," Working Papers halshs-02512932, HAL.
    12. Komro, K.A. & Livingston, M.D. & Markowitz, S. & Wagenaar, A.C., 2016. "The effect of an increased minimum wage on infant mortality and birth weight," American Journal of Public Health, American Public Health Association, vol. 106(8), pages 1514-1516.
    13. Michael J. Weiss & Howard S. Bloom & Thomas Brock, 2014. "A Conceptual Framework For Studying The Sources Of Variation In Program Effects," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(3), pages 778-808, June.
    14. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    15. Emily Sama-Miller & Lauren Akers & Andrea Mraz-Esposito & Rebecca Coughlin & Marykate Zukiewicz, "undated". "Home Visiting Evidence of Effectiveness Review: Executive Summary," Mathematica Policy Research Reports ab92c1547bc142a6815d64f33, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John List, 2021. "The Voltage Effect in Behavioral Economics," Artefactual Field Experiments 00733, The Field Experiments Website.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John A. List, 2024. "Optimally generate policy-based evidence before scaling," Nature, Nature, vol. 626(7999), pages 491-499, February.
    2. Omar Al-Ubaydli & John List & Dana Suskind, 2019. "The science of using science: Towards an understanding of the threats to scaling experiments," Artefactual Field Experiments 00670, The Field Experiments Website.
    3. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. Goldberg, Jessica & Bumgarner, Erin & Jacobs, Francine, 2016. "Measuring program- and individual-level fidelity in a home visiting program for adolescent parents," Evaluation and Program Planning, Elsevier, vol. 55(C), pages 163-173.
    6. Julian, Megan M. & Muzik, Maria & Jester, Jennifer M. & Handelzalts, Jonathan & Erickson, Nora & Stringer, Marissa & Brophy-Herb, Holly & Ribaudo, Julie & Huth-Bocks, Alissa & Lawler, Jamie & Stacks, , 2021. "Relationships heal: Reducing harsh parenting and child abuse potential with relationship-based parent-infant home visiting," Children and Youth Services Review, Elsevier, vol. 128(C).
    7. Wen, Xiaoli & Korfmacher, Jon & Hans, Sydney L., 2016. "Change over time in young mothers' engagement with a community-based doula home visiting program," Children and Youth Services Review, Elsevier, vol. 69(C), pages 116-126.
    8. Brian Goesling & Sarah Oberlander & Lisa Trivits, 2017. "High-Stakes Systematic Reviews," Evaluation Review, , vol. 41(1), pages 27-49, February.
    9. Jan-Emmanuel De Neve & Clément Imbert & Johannes Spinnewijn & Teodora Tsankova & Maarten Luts, 2021. "How to Improve Tax Compliance? Evidence from Population-Wide Experiments in Belgium," Journal of Political Economy, University of Chicago Press, vol. 129(5), pages 1425-1463.
    10. Andrew Dustan & Stanislao Maldonado & Juan Manuel Hernandez-Agramonte, 2018. "Motivating bureaucrats with non-monetary incentives when state capacity is weak: Evidence from large-scale field experiments in Peru," Working Papers 136, Peruvian Economic Association.
    11. Mariella Gonzales & Gianmarco León-Ciliotta & Luis R. Martínez, 2022. "How Effective Are Monetary Incentives to Vote? Evidence from a Nationwide Policy," American Economic Journal: Applied Economics, American Economic Association, vol. 14(1), pages 293-326, January.
    12. Rebecca A. Maynard, 2018. "The Role of Federal Agencies in Creating and Administering Evidence-Based Policies," The ANNALS of the American Academy of Political and Social Science, , vol. 678(1), pages 134-144, July.
    13. Omar Al‐Ubaydli & John A. List & Dana Suskind, 2020. "2017 Klein Lecture: The Science Of Using Science: Toward An Understanding Of The Threats To Scalability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 61(4), pages 1387-1409, November.
    14. Orla Doyle, 2017. "The First 2,000 Days and Child Skills: Evidence from a Randomized Experiment of Home Visiting," Working Papers 2017-054, Human Capital and Economic Opportunity Working Group.
    15. Jessica Harding & Jean Knab & Susan Zief & Kevin Kelly & Diana McCallum, "undated". "A Systematic Review of Programs to Promote Aspects of Teen Parents’ Self-Sufficiency: Supporting Educational Outcomes and Healthy Birth Spacing," Mathematica Policy Research Reports 05e656f1b4a54dae83654795b, Mathematica Policy Research.
    16. Gabriella Conti, 2013. "The Developmental Origins of Health Inequality," Research on Economic Inequality, in: Health and Inequality, volume 21, pages 285-309, Emerald Group Publishing Limited.
    17. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    18. Schreier, Alayna & McCoy, Kelsey & Flood, Mary Fran & Wilcox, Brian L. & Hansen, David J., 2020. "Early Head Start service use by families with court-substantiated maltreatment," Children and Youth Services Review, Elsevier, vol. 108(C).
    19. Samuel Berlinski & Norbert Schady, 2015. "Daycare Services: It’s All about Quality," Palgrave Macmillan Books, in: Samuel Berlinski & Norbert Schady (ed.), The Early Years, chapter 4, pages 91-119, Palgrave Macmillan.
    20. Jules Gazeaud & Claire Ricard, 2021. "Conditional cash transfers and the learning crisis: evidence from Tayssir scale-up in Morocco," NOVAFRICA Working Paper Series wp2102, Universidade Nova de Lisboa, Nova School of Business and Economics, NOVAFRICA.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:feb:natura:00741. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: David Franks (email available below). General contact details of provider: http://www.fieldexperiments.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.