IDEAS home Printed from https://ideas.repec.org/p/zbw/rwirep/790.html
   My bibliography  Save this paper

Do planning prompts increase educational success? Evidence from randomized controlled trials in MOOCs

Author

Listed:
  • Andor, Mark A.
  • Fels, Katja M.
  • Renz, Jan
  • Rzepka, Sylvi

Abstract

Massive Open Online Courses are a promising educational innovation. Yet, they suffer from high drop-out rates. As a remedy, we propose a planning prompt and test its effect on course completion and further outcomes such as course engagement and satisfaction in four large-scale randomized controlled trials. The results reveal an overall null effect on the completion rate, ruling out effect sizes beyond the [-7%, 3%] interval. However, this overall effect masks heterogeneity across and within courses: In one course the planning prompt increases course completion by 19%, highlighting the importance of replications in slightly different contexts. Using random causal forests, we also reveal tendencies for differential effects by subgroups. Better targeting could hence improve the effectiveness of planning prompts in online learning.

Suggested Citation

  • Andor, Mark A. & Fels, Katja M. & Renz, Jan & Rzepka, Sylvi, 2018. "Do planning prompts increase educational success? Evidence from randomized controlled trials in MOOCs," Ruhr Economic Papers 790, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    • Handle: RePEc:zbw:rwirep:790
    DOI: 10.4419/86788918
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/191803/1/1048274179.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.4419/86788918?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Heather Royer & Mark Stehr & Justin Sydnor, 2015. "Incentives, Commitments, and Habit Formation in Exercise: Evidence from a Field Experiment with Workers at a Fortune-500 Company," American Economic Journal: Applied Economics, American Economic Association, vol. 7(3), pages 51-84, July.
    2. Damgaard, Mette Trier & Gravert, Christina, 2018. "The hidden costs of nudging: Experimental evidence from reminders in fundraising," Journal of Public Economics, Elsevier, vol. 157(C), pages 15-26.
    3. Eva Vivalt, 2015. "Heterogeneous Treatment Effects in Impact Evaluation," American Economic Review, American Economic Association, vol. 105(5), pages 467-470, May.
    4. Steven D. Levitt & John A. List & Susanne Neckermann & Sally Sadoff, 2016. "The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance," American Economic Journal: Economic Policy, American Economic Association, vol. 8(4), pages 183-219, November.
    5. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    6. Patterson, Richard W., 2018. "Can behavioral tools improve online student outcomes? Experimental evidence from a massive open online course," Journal of Economic Behavior & Organization, Elsevier, vol. 153(C), pages 293-321.
    7. Adam M. Lavecchia & Heidi Liu & Philip Oreopoulos, 2014. "Behavioral Economics of Education: Progress and Possibilities," NBER Working Papers 20609, National Bureau of Economic Research, Inc.
    8. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    9. Annamaria Lusardi & Punam Anand Keller & Adam M. Keller, 2009. "New Ways to Make People Save: A Social Marketing Approach," NBER Working Papers 14715, National Bureau of Economic Research, Inc.
    10. John Beshears & Katherine L. Milkman & Joshua Schwartzstein, 2016. "Beyond Beta-Delta: The Emerging Economics of Personal Plans," American Economic Review, American Economic Association, vol. 106(5), pages 430-434, May.
    11. O'Donoghue, Ted & Rabin, Matthew, 2008. "Procrastination on long-term projects," Journal of Economic Behavior & Organization, Elsevier, vol. 66(2), pages 161-175, May.
    12. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    13. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    14. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    15. Omar Al-Ubaydli & John A. List & Dana L. Suskind, 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results," American Economic Review, American Economic Association, vol. 107(5), pages 282-286, May.
    16. Katherine L. Milkman & John Beshears & James J. Choi & David Laibson & Brigitte C. Madrian, 2011. "Using Implementation Intentions Prompts to Enhance Influenza Vaccination Rates," NBER Working Papers 17183, National Bureau of Economic Research, Inc.
    17. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    18. Damgaard, Mette Trier & Nielsen, Helena Skyt, 2018. "Nudging in education," Economics of Education Review, Elsevier, vol. 64(C), pages 313-342.
    19. Lavecchia, A.M. & Liu, H. & Oreopoulos, P., 2016. "Behavioral Economics of Education," Handbook of the Economics of Education,, Elsevier.
    20. David P. Byrne & Andrea La Nauze & Leslie A. Martin, 2018. "Tell Me Something I Don’t Already Know: Informedness and the Impact of Information Programs," The Review of Economics and Statistics, MIT Press, vol. 100(3), pages 510-527, July.
    21. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    22. Alexander K. Koch & Julia Nafziger, 2011. "Self‐regulation through Goal Setting," Scandinavian Journal of Economics, Wiley Blackwell, vol. 113(1), pages 212-227, March.
    23. Carrera, Mariana & Royer, Heather & Stehr, Mark & Sydnor, Justin & Taubinsky, Dmitry, 2018. "The limits of simple implementation intentions: Evidence from a field experiment on making plans to exercise," Journal of Health Economics, Elsevier, vol. 62(C), pages 95-104.
    24. Nadja Dwenger & Henrik Kleven & Imran Rasul & Johannes Rincke, 2016. "Extrinsic and Intrinsic Motivations for Tax Compliance: Evidence from a Field Experiment in Germany," American Economic Journal: Economic Policy, American Economic Association, vol. 8(3), pages 203-232, August.
    25. repec:feb:artefa:0087 is not listed on IDEAS
    26. Omar Al-Ubaydli & John A. List & Danielle LoRe & Dana Suskind, 2017. "Scaling for Economists: Lessons from the Non-Adherence Problem in the Medical Literature," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 125-144, Fall.
    27. Koch, Alexander & Nafziger, Julia & Nielsen, Helena Skyt, 2015. "Behavioral economics of education," Journal of Economic Behavior & Organization, Elsevier, vol. 115(C), pages 3-17.
    28. Philip Oreopoulos & Richard W. Patterson & Uros Petronijevic & Nolan G. Pope, 2018. "When Studying and Nudging Don’t Go as Planned: Unsuccessful Attempts to Help Traditional and Online College Students," NBER Working Papers 25036, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andor, Mark A. & Gerster, Andreas & Peters, Jörg, 2022. "Information campaigns for residential energy conservation," European Economic Review, Elsevier, vol. 144(C).
    2. Andor, Mark A. & Gerster, Andreas & Peters, Jörg & Schmidt, Christoph M., 2020. "Social Norms and Energy Conservation Beyond the US," Journal of Environmental Economics and Management, Elsevier, vol. 103(C).
    3. Damgaard, Mette Trier & Nielsen, Helena Skyt, 2018. "Nudging in education," Economics of Education Review, Elsevier, vol. 64(C), pages 313-342.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. Wagner, Valentin, 2016. "Seeking risk or answering smart? Framing in elementary schools," DICE Discussion Papers 227, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    6. John A. List, 2024. "Optimally generate policy-based evidence before scaling," Nature, Nature, vol. 626(7999), pages 491-499, February.
    7. Damon Clark & David Gill & Victoria Prowse & Mark Rush, 2020. "Using Goals to Motivate College Students: Theory and Evidence From Field Experiments," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 648-663, October.
    8. Islam, Asad & Kwon, Sungoh & Masood, Eema & Prakash, Nishith & Sabarwal, Shwetlena & Saraswat, Deepak, 2020. "When Goal-Setting Forges Ahead but Stops Short," GLO Discussion Paper Series 526, Global Labor Organization (GLO).
    9. Burgess, Simon & Metcalfe, Robert & Sadoff, Sally, 2021. "Understanding the response to financial and non-financial incentives in education: Field experimental evidence using high-stakes assessments," Economics of Education Review, Elsevier, vol. 85(C).
    10. Oliver Himmler & Robert Jäckle & Philipp Weinschenk, 2019. "Soft Commitments, Reminders, and Academic Performance," American Economic Journal: Applied Economics, American Economic Association, vol. 11(2), pages 114-142, April.
    11. Andor, Mark A. & Fels, Katja M., 2018. "Behavioral Economics and Energy Conservation – A Systematic Review of Non-price Interventions and Their Causal Effects," Ecological Economics, Elsevier, vol. 148(C), pages 178-210.
    12. Eva m. Berger & Guenther Koenig & Henning Müller & Felix Schmidt & Daniel Schunk, 2017. "Self-Regulation Training and Job Search Effort: A Natural Field Experiment within an Active Labor Market Program," Working Papers 1712, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.
    13. Omar Al-Ubaydli & John List & Dana Suskind, 2019. "The science of using science: Towards an understanding of the threats to scaling experiments," Artefactual Field Experiments 00670, The Field Experiments Website.
    14. Brigitte C. Madrian, 2014. "Applying Insights from Behavioral Economics to Policy Design," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 663-688, August.
    15. Castleman, Benjamin L. & Murphy, Francis X. & Patterson, Richard & Skimmyhorn, William L., 2019. "Active Choice Framing and Intergenerational Education Benefits: Evidence from the Field," IZA Discussion Papers 12523, Institute of Labor Economics (IZA).
    16. Eva M. Berger & Günther König & Henning Mueller & Felix Schmidt & Daniel Schunk, 2016. "Self-Regulation Training, Labor Market Reintegration of Unemployed Individuals, and Locus of Control - Evidence from a Natural Field Experiment," CESifo Working Paper Series 6246, CESifo.
    17. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    18. van Lent, Max & Souverijn, Michiel, 2020. "Goal setting and raising the bar: A field experiment," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 87(C).
    19. Damgaard, Mette Trier, 2021. "A decade of nudging: What have we learned?," Nationaløkonomisk tidsskrift, Nationaløkonomisk Forening, vol. 2021(1), pages 1-21.
    20. Christina Gravert & Linus Olsson Collentine, 2019. "When nudges aren't enough: Incentives and habit formation in public transport usage," CEBI working paper series 19-10, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).

    More about this item

    Keywords

    massive open online courses; planning prompt; behavioral economics; natural field experiment;
    All these keywords.

    JEL classification:

    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I29 - Health, Education, and Welfare - - Education - - - Other
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:rwirep:790. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/rwiesde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.