IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i6p969-999.html

Assessing the Impact of a University Transition Online Course on Student Continuation Using Statistical Matching Methods

Author

Listed:
  • Billy Wong
  • Lydia Fletcher

Abstract

This study demonstrates how to evaluate a university-wide online course designed to support student transition into university by using Propensity Score Matching (PSM) and Doubly Robust Estimation (DRE). Using data from seven academic years, from 2016/17 to 2022/23, with more than 28,000 students, we examine whether enrolment in this optional pre-arrival course affects first-year pass rates. We also conducted additional analyses to compare outcomes from the year before and after the course’s implementation, as well as to examine these patterns across recent cohorts to potentially account for contextual changes over time. Results indicate that enrolled students show a 6.2 percentage point increase in the likelihood of passing Year 1, controlling for factors including sex, domicile, age, ethnicity, disability and socioeconomic status. We demonstrate how utilising existing institutional data can potentially strengthen evidence of impact for centralised initiatives and conclude with reflections on the use of such institutional data and matching techniques and their viability for future evaluations.

Suggested Citation

  • Billy Wong & Lydia Fletcher, 2025. "Assessing the Impact of a University Transition Online Course on Student Continuation Using Statistical Matching Methods," Evaluation Review, , vol. 49(6), pages 969-999, December.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:969-999
    DOI: 10.1177/0193841X251339686
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X251339686
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X251339686?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Susan M. Dynarski, 2003. "Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and Completion," American Economic Review, American Economic Association, vol. 93(1), pages 279-288, March.
    2. S. Derya Uysal, 2015. "Doubly Robust Estimation of Causal Effects with Multivalued Treatments: An Application to the Returns to Schooling," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(5), pages 763-786, August.
    3. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    4. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    5. King, Gary & Nielsen, Richard, 2019. "Why Propensity Scores Should Not Be Used for Matching," Political Analysis, Cambridge University Press, vol. 27(4), pages 435-454, October.
    6. Jill L. Caviglia-Harris, 2022. "Community is key: estimating the impact of living learning communities on college retention and GPA," Education Economics, Taylor & Francis Journals, vol. 30(2), pages 173-190, March.
    7. Martha Cecilia Bottia & Elizabeth Stearns & Roslyn Arlin Mickelson & Stephanie Moller & Cayce Jamil, 2020. "The Importance of Community Colleges in Students’ Choice to Major in STEM," The Journal of Higher Education, Taylor & Francis Journals, vol. 91(7), pages 1116-1148, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huang, Wei & Li, Fan & Liao, Xiaowei & Hu, Pingping, 2018. "More money, better performance? The effects of student loans and need-based grants in China's higher education," China Economic Review, Elsevier, vol. 51(C), pages 208-227.
    2. Daniel Litwok, 2023. "Estimating the Impact of Emergency Assistance on Educational Progress for Low-Income Adults: Experimental and Nonexperimental Evidence," Evaluation Review, , vol. 47(2), pages 231-263, April.
    3. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    4. Akoh Fabien Yao & Maxime Sèbe & Laura Recuero Virto & Abdelhak Nassiri & Hervé Dumez, 2024. "The effect of LNG bunkering on port competitiveness using multilevel data analysis [L'effet du soutage par GNL sur la compétitivité des ports à l'aide de l'analyse de données à plusieurs niveaux]," Post-Print hal-04611804, HAL.
    5. Fossen, Frank & Simmler, Martin, 2012. "Differential taxation and firms' financial leverage: Evidence from the introduction of a flat tax on interest income," Discussion Papers 2012/4, Free University Berlin, School of Business & Economics.
    6. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    7. Donatien Eze Eze, 2019. "Microfinance programs and domestic violence in northern Cameroon; the case of the Familial Rural Income Improvement Program," Review of Economics of the Household, Springer, vol. 17(3), pages 947-967, September.
    8. Ito, Junichi & Feuer, Hart N. & Kitano, Shinichi & Asahi, Haruka, 2019. "Assessing the effectiveness of Japan's community-based direct payment scheme for hilly and mountainous areas," Ecological Economics, Elsevier, vol. 160(C), pages 62-75.
    9. Farrell, Max H., 2015. "Robust inference on average treatment effects with possibly more covariates than observations," Journal of Econometrics, Elsevier, vol. 189(1), pages 1-23.
    10. Tohari, Achmad & Parsons, Christopher & Rammohan, Anu, 2019. "Targeting poverty under complementarities: Evidence from Indonesia's unified targeting system," Journal of Development Economics, Elsevier, vol. 140(C), pages 127-144.
    11. Seonho Shin, 2022. "Evaluating the Effect of the Matching Grant Program for Refugees: An Observational Study Using Matching, Weighting, and the Mantel-Haenszel Test," Journal of Labor Research, Springer, vol. 43(1), pages 103-133, March.
    12. Zetterqvist, Johan & Waernbaum, Ingeborg, 2020. "Semi-parametric estimation of multi-valued treatment effects for the treated:estimating equations and sandwich estimators," Working Paper Series 2020:4, IFAU - Institute for Evaluation of Labour Market and Education Policy.
    13. Fatih Unlu & Douglas Lee Lauen & Sarah Crittenden Fuller & Tiffany Berglund & Elc Estrera, 2021. "Can Quasi‐Experimental Evaluations That Rely On State Longitudinal Data Systems Replicate Experimental Results?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(2), pages 572-613, March.
    14. Verena Lauber & Johanna Storck, 2016. "Helping with the Kids? How Family-Friendly Workplaces Affect Parental Well-Being and Behavior," SOEPpapers on Multidisciplinary Panel Data Research 883, DIW Berlin, The German Socio-Economic Panel (SOEP).
    15. Md Abdul Wasi & Thu Phuong Pham & Ralf Zurbruegg, 2023. "Bank systemic risk: An analysis of the sovereign rating ceiling policy and rating downgrades," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 50(1-2), pages 411-440, January.
    16. Shu Yang & Yunshu Zhang, 2023. "Multiply robust matching estimators of average and quantile treatment effects," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 50(1), pages 235-265, March.
    17. Coulibaly, Yacouba, 2025. "The effects of resource-backed loans on deforestation: Evidence from developing countries," World Development, Elsevier, vol. 188(C).
    18. Yacouba Coulibaly, 2023. "Can Resource-backed Loans Mitigate Climate Change ?," Working Papers hal-04072352, HAL.
    19. Lars Thiel, 2015. "Leave the Drama on the Stage: The Effect of Cultural Participation on Health," SOEPpapers on Multidisciplinary Panel Data Research 767, DIW Berlin, The German Socio-Economic Panel (SOEP).
    20. Matthew Blackwell & Anton Strezhnev, 2022. "Telescope matching for reducing model dependence in the estimation of the effects of time‐varying treatments: An application to negative advertising," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(1), pages 377-399, January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:969-999. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.