IDEAS home Printed from https://ideas.repec.org/p/cep/cepdps/dp2176.html

From pilot to policy: experimental evidence from Scaling online tutoring

Author

Listed:
  • Maria Calle
  • Lucas Gortazar
  • Maria Hernandez-de-Benito
  • Claudia Hupkau
  • Teresa Molina-Millan
  • Antonio Roldan Mones

Abstract

Can effective tutoring programs survive the transition from pilot to scale, and from researcher to government implementers? We address this question using a randomized controlled trial of an online mathematics tutoring program scaled from a successful pilot and implemented entirely by regional education authorities in Spain. Critically, tutoring was delivered by interim public-school teachers-qualified educators already integrated into the government's teacher registry-rather than specially recruited tutors, offering a test of whether effects persist under a lower-cost, readily scalable staffing model. Assignment to tutoring increased end-of-year grades by 0.15σ and standardized math test scores by 0.11σ - approximately one-third of the original pilot effects. Treated students also reported improved confidence in their math abilities and reduced math anxiety. An experimental socio-emotional training module for tutors improved affective outcomes but did not enhance academic gains. The findings demonstrate that online tutoring can generate meaningful benefits at scale under government implementation, while providing realistic benchmarks for the "voltage drop" policymakers should anticipate when adopting evidence-based interventions.

Suggested Citation

  • Maria Calle & Lucas Gortazar & Maria Hernandez-de-Benito & Claudia Hupkau & Teresa Molina-Millan & Antonio Roldan Mones, 2026. "From pilot to policy: experimental evidence from Scaling online tutoring," CEP Discussion Papers dp2176, Centre for Economic Performance, LSE.
    • Handle: RePEc:cep:cepdps:dp2176
    as

    Download full text from publisher

    File URL: https://cep.lse.ac.uk/pubs/download/dp2176.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    2. Justman, Moshe, 2018. "Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction," European Journal of Political Economy, Elsevier, vol. 54(C), pages 167-174.
    3. Eric P. Bettinger & Lindsay Fox & Susanna Loeb & Eric S. Taylor, 2017. "Virtual Classrooms: How Online College Courses Affect Student Success," American Economic Review, American Economic Association, vol. 107(9), pages 2855-2875, September.
    4. Matthew A. Kraft & John A. List & Jeffrey A. Livingston & Sally Sadoff, 2022. "Online Tutoring by College Volunteers: Experimental Evidence from a Pilot Program," AEA Papers and Proceedings, American Economic Association, vol. 112, pages 614-618, May.
    5. David S. Lee, 2009. "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(3), pages 1071-1102.
    6. Maya Escueta & Andre Joshua Nickow & Philip Oreopoulos & Vincent Quan, 2020. "Upgrading Education with Technology: Insights from Experimental Research," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 897-996, December.
    7. Eva Vivalt, 2020. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    8. Bold, Tessa & Kimenyi, Mwangi & Mwabu, Germano & Ng’ang’a, Alice & Sandefur, Justin, 2018. "Experimental evidence on scaling up education reforms in Kenya," Journal of Public Economics, Elsevier, vol. 168(C), pages 1-20.
    9. Michela Carlana & Eliana La Ferrara, 2025. "Apart but Connected: Online Tutoring, Cognitive Outcomes, and Soft Skills," American Economic Review, American Economic Association, vol. 115(10), pages 3487-3513, October.
    10. Omar Al-Ubaydli & John A. List & Dana L. Suskind, 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results," American Economic Review, American Economic Association, vol. 107(5), pages 282-286, May.
    11. Hardt, David & Nagler, Markus & Rincke, Johannes, 2023. "Tutoring in (online) higher education: Experimental evidence," Economics of Education Review, Elsevier, vol. 92(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gortazar, Lucas & Hupkau, Claudia & Roldán-Monés, Antonio, 2024. "Online tutoring works: Experimental evidence from a program with vulnerable children," Journal of Public Economics, Elsevier, vol. 232(C).
    2. Biroli, Pietro & Di Girolamo, Amalia & Sorrenti, Giuseppe & Totarelli, Maddalena, 2025. "Talent Is Everywhere, Opportunity Is Not: Online Role Model Mentoring and Students’ Aspirations," IZA Discussion Papers 18325, IZA Network @ LISER.
    3. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    4. Albornoz, Facundo & Almeyda Torres, Gonzalo & Lombardi, María & Oubiña, Victoria & Lobaton, Pablo Zoido, 2026. "Remote tutoring in Latin America," Journal of Development Economics, Elsevier, vol. 179(C).
    5. Ásgeirsdóttir, Tinna Laufey & Francesconi, Marco & Jóhannsdóttir, Ásthildur M. & Zoega, Gylfi, 2026. "How home exams and peers affect college grades in unprecedented times," European Economic Review, Elsevier, vol. 183(C).
    6. Jason T. Kerwin & Rebecca L. Thornton, 2021. "Making the Grade: The Sensitivity of Education Program Effectiveness to Input Choices and Outcome Measures," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 251-264, May.
    7. Bonesrønning, Hans & Finseraas, Henning & Hardoy, Ines & Iversen, Jon Marius Vaag & Nyhus, Ole Henning & Opheim, Vibeke & Salvanes, Kari Vea & Sandsør, Astrid Marie Jorde & Schøne, Pål, 2022. "Small-group instruction to improve student performance in mathematics in early grades: Results from a randomized field experiment," Journal of Public Economics, Elsevier, vol. 216(C).
    8. Holla,Alaka & Bendini,Maria Magdalena & Dinarte Diaz,Lelys Ileana & Trako,Iva, 2021. "Is Investment in Preprimary Education Too Low ? Lessons from (Quasi) ExperimentalEvidence across Countries," Policy Research Working Paper Series 9723, The World Bank.
    9. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    10. repec:osf:osfxxx:nwp8k_v1 is not listed on IDEAS
    11. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    12. Ferman, Bruno & Lima, Lycia & Riva, Flávio, 2021. "Artificial Intelligence, Teacher Tasks and Individualized Pedagogy," SocArXiv qw249, Center for Open Science.
    13. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.
    14. Islam, Asad & Kwon, Sungoh & Masood, Eema & Prakash, Nishith & Sabarwal, Shwetlena & Saraswat, Deepak, 2020. "When Goal-Setting Forges Ahead but Stops Short," GLO Discussion Paper Series 526, Global Labor Organization (GLO).
    15. Grosch, Kerstin & Haeckl, Simone & Kocher , Martin G., 2022. "Closing the gender STEM gap. A large-scale randomized-controlled trial in elementary schools," UiS Working Papers in Economics and Finance 2022/4, University of Stavanger.
    16. Eliana Carranza & Robert Garlick & Kate Orkin & Neil Rankin, 2020. "Job Search and Hiring with Two-sided Limited Information about Workseekers’ Skills," CSAE Working Paper Series 2020-10, Centre for the Study of African Economies, University of Oxford.
    17. Sun, Yue & Tang, Can & Zhao, Zhong, 2024. "Does online education magnify educational inequalities? Evidence from the COVID-19 pandemic," China Economic Review, Elsevier, vol. 88(C).
    18. Werthschulte, Madeline, 2023. "Present focus and billing systems: Testing ‘pay-as-you-go’ vs. ‘pay-later’," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 108-121.
    19. Ali, Ayesha & Qazi, Ihsan Ayyub, 2023. "Countering misinformation on social media through educational interventions: Evidence from a randomized experiment in Pakistan," Journal of Development Economics, Elsevier, vol. 163(C).
    20. Lisa Barrow & Lashawn Richburg-Hayes & Cecilia Elena Rouse & Thomas Brock, 2014. "Paying for Performance: The Education Impacts of a Community College Scholarship Program for Low-Income Adults," Journal of Labor Economics, University of Chicago Press, vol. 32(3), pages 563-599.
    21. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cep:cepdps:dp2176. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://cep.lse.ac.uk/_new/publications/discussion-papers/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.