IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i6p969-999.html

Assessing the Impact of a University Transition Online Course on Student Continuation Using Statistical Matching Methods

Author

Listed:
  • Billy Wong
  • Lydia Fletcher

Abstract

This study demonstrates how to evaluate a university-wide online course designed to support student transition into university by using Propensity Score Matching (PSM) and Doubly Robust Estimation (DRE). Using data from seven academic years, from 2016/17 to 2022/23, with more than 28,000 students, we examine whether enrolment in this optional pre-arrival course affects first-year pass rates. We also conducted additional analyses to compare outcomes from the year before and after the course’s implementation, as well as to examine these patterns across recent cohorts to potentially account for contextual changes over time. Results indicate that enrolled students show a 6.2 percentage point increase in the likelihood of passing Year 1, controlling for factors including sex, domicile, age, ethnicity, disability and socioeconomic status. We demonstrate how utilising existing institutional data can potentially strengthen evidence of impact for centralised initiatives and conclude with reflections on the use of such institutional data and matching techniques and their viability for future evaluations.

Suggested Citation

  • Billy Wong & Lydia Fletcher, 2025. "Assessing the Impact of a University Transition Online Course on Student Continuation Using Statistical Matching Methods," Evaluation Review, , vol. 49(6), pages 969-999, December.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:969-999
    DOI: 10.1177/0193841X251339686
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X251339686
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X251339686?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    2. King, Gary & Nielsen, Richard, 2019. "Why Propensity Scores Should Not Be Used for Matching," Political Analysis, Cambridge University Press, vol. 27(4), pages 435-454, October.
    3. Susan M. Dynarski, 2003. "Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and Completion," American Economic Review, American Economic Association, vol. 93(1), pages 279-288, March.
    4. S. Derya Uysal, 2015. "Doubly Robust Estimation of Causal Effects with Multivalued Treatments: An Application to the Returns to Schooling," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(5), pages 763-786, August.
    5. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    6. Jill L. Caviglia-Harris, 2022. "Community is key: estimating the impact of living learning communities on college retention and GPA," Education Economics, Taylor & Francis Journals, vol. 30(2), pages 173-190, March.
    7. Martha Cecilia Bottia & Elizabeth Stearns & Roslyn Arlin Mickelson & Stephanie Moller & Cayce Jamil, 2020. "The Importance of Community Colleges in Students’ Choice to Major in STEM," The Journal of Higher Education, Taylor & Francis Journals, vol. 91(7), pages 1116-1148, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huang, Wei & Li, Fan & Liao, Xiaowei & Hu, Pingping, 2018. "More money, better performance? The effects of student loans and need-based grants in China's higher education," China Economic Review, Elsevier, vol. 51(C), pages 208-227.
    2. Daniel Litwok, 2023. "Estimating the Impact of Emergency Assistance on Educational Progress for Low-Income Adults: Experimental and Nonexperimental Evidence," Evaluation Review, , vol. 47(2), pages 231-263, April.
    3. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    4. Akoh Fabien Yao & Maxime Sèbe & Laura Recuero Virto & Abdelhak Nassiri & Hervé Dumez, 2024. "The effect of LNG bunkering on port competitiveness using multilevel data analysis [L'effet du soutage par GNL sur la compétitivité des ports à l'aide de l'analyse de données à plusieurs niveaux]," Post-Print hal-04611804, HAL.
    5. Eliane Badaoui & Eleonora Matteazzi & Vincenzo Prete, 2024. "The role of non‐base compensation in explaining the motherhood wage gap: Evidence from Italy," Kyklos, Wiley Blackwell, vol. 77(4), pages 873-894, November.
    6. Hao Cheng & Ying Wei, 2018. "A fast imputation algorithm in quantile regression," Computational Statistics, Springer, vol. 33(4), pages 1589-1603, December.
    7. Fossen, Frank & Simmler, Martin, 2012. "Differential taxation and firms' financial leverage: Evidence from the introduction of a flat tax on interest income," Discussion Papers 2012/4, Free University Berlin, School of Business & Economics.
    8. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    9. Marianne BLÉHAUT & Xavier D'HAULTFOEUILLE & Jérémy L'HOUR & Alexandre B. TSYBAKOV, 2020. "An alternative to synthetic control for models with many covariates under sparsity," Working Papers 2020-17, Center for Research in Economics and Statistics.
    10. Aloyce R. Kaliba & Anne G. Gongwe & Kizito Mazvimavi & Ashagre Yigletu, 2021. "Impact of Adopting Improved Seeds on Access to Broader Food Groups Among Small-Scale Sorghum Producers in Tanzania," SAGE Open, , vol. 11(1), pages 21582440209, January.
    11. Donatien Eze Eze, 2019. "Microfinance programs and domestic violence in northern Cameroon; the case of the Familial Rural Income Improvement Program," Review of Economics of the Household, Springer, vol. 17(3), pages 947-967, September.
    12. Xuelin Yang & Licong Lin & Susan Athey & Michael I. Jordan & Guido W. Imbens, 2025. "Cross-Validated Causal Inference: a Modern Method to Combine Experimental and Observational Data," Papers 2511.00727, arXiv.org.
    13. Ito, Junichi & Feuer, Hart N. & Kitano, Shinichi & Asahi, Haruka, 2019. "Assessing the effectiveness of Japan's community-based direct payment scheme for hilly and mountainous areas," Ecological Economics, Elsevier, vol. 160(C), pages 62-75.
    14. Farrell, Max H., 2015. "Robust inference on average treatment effects with possibly more covariates than observations," Journal of Econometrics, Elsevier, vol. 189(1), pages 1-23.
    15. Manuel Salas-Velasco, 2025. "Non-experimental impact estimates of school financial education exposure on financial literacy outcomes," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 42(2), pages 379-403, July.
    16. Tohari, Achmad & Parsons, Christopher & Rammohan, Anu, 2019. "Targeting poverty under complementarities: Evidence from Indonesia's unified targeting system," Journal of Development Economics, Elsevier, vol. 140(C), pages 127-144.
    17. Seonho Shin, 2022. "Evaluating the Effect of the Matching Grant Program for Refugees: An Observational Study Using Matching, Weighting, and the Mantel-Haenszel Test," Journal of Labor Research, Springer, vol. 43(1), pages 103-133, March.
    18. Mehmet Ugur & Eshref Trushin, 2023. "Information asymmetry, risk aversion and R&D subsidies: effect-size heterogeneity and policy conundrums," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 32(8), pages 1190-1215, November.
    19. Verena Lauber & Johanna Storck, 2016. "Helping with the Kids? How Family-Friendly Workplaces Affect Parental Well-Being and Behavior," Discussion Papers of DIW Berlin 1630, DIW Berlin, German Institute for Economic Research.
    20. Jan Marcus, 2014. "Does Job Loss Make You Smoke and Gain Weight?," Economica, London School of Economics and Political Science, vol. 81(324), pages 626-648, October.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:969-999. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.