IDEAS home Printed from https://ideas.repec.org/a/spr/reihed/v62y2021i8d10.1007_s11162-021-09632-0.html
   My bibliography  Save this article

Improving Institutional Evaluation Methods: Comparing Three Evaluations Using PSM, Exact and Coarsened Exact Matching

Author

Listed:
  • Bob Blankenberger

    (University of Illinois at Springfield)

  • Sophia Gehlhausen Anderson

    (Illinois Board of Higher Education
    University of Illinois at Springfield)

  • Eric Lichtenberger

    (Illinois Board of Higher Education)

Abstract

Policymakers and institutional leaders in higher education too often make decisions based on descriptive data analyses or even anecdote when better analysis options could produce more nuanced and more valuable results. Employing the setting of higher education program evaluation at a midwestern regional public university, for this study we compared analysis approaches using basic descriptive analyses, regression, standard propensity score matching (PSM), and a mixture of PSM with continuous variables, coarsened exact matching, and exact matching on categorical variables. We used three examples of program evaluations: a freshman seminar, an upper division general education program intended to improve cultural awareness and respect for diverse groups, and multiple living learning communities. We describe how these evaluations were conducted, compare the different results for each type of method employed, and discuss the strengths and weaknesses of each in the context of program evaluation.

Suggested Citation

  • Bob Blankenberger & Sophia Gehlhausen Anderson & Eric Lichtenberger, 2021. "Improving Institutional Evaluation Methods: Comparing Three Evaluations Using PSM, Exact and Coarsened Exact Matching," Research in Higher Education, Springer;Association for Institutional Research, vol. 62(8), pages 1248-1275, December.
    • Handle: RePEc:spr:reihed:v:62:y:2021:i:8:d:10.1007_s11162-021-09632-0
    DOI: 10.1007/s11162-021-09632-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11162-021-09632-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11162-021-09632-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    2. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marco Morucci & Md. Noor-E-Alam & Cynthia Rudin, 2022. "A Robust Approach to Quantifying Uncertainty in Matching Problems of Causal Inference," INFORMS Joural on Data Science, INFORMS, vol. 1(2), pages 156-171, October.
    2. Jason J. Sauppe & Sheldon H. Jacobson & Edward C. Sewell, 2014. "Complexity and Approximation Results for the Balance Optimization Subset Selection Model for Causal Inference in Observational Studies," INFORMS Journal on Computing, INFORMS, vol. 26(3), pages 547-566, August.
    3. Jones, Andrew M. & Rice, Nigel & Zantomio, Francesca, 2020. "Acute health shocks and labour market outcomes: Evidence from the post crash era," Economics & Human Biology, Elsevier, vol. 36(C).
    4. Jason J. Sauppe & Sheldon H. Jacobson, 2017. "The role of covariate balance in observational studies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 64(4), pages 323-344, June.
    5. Adeola Oyenubi & Martin Wittenberg, 2021. "Does the choice of balance-measure matter under genetic matching?," Empirical Economics, Springer, vol. 61(1), pages 489-502, July.
    6. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    7. Francesco Lamperti & Roberto Mavilia & Simona Castellini, 2017. "The role of Science Parks: a puzzle of growth, innovation and R&D investments," The Journal of Technology Transfer, Springer, vol. 42(1), pages 158-183, February.
    8. Hao Chen & Dylan S. Small, 2022. "New multivariate tests for assessing covariate balance in matched observational studies," Biometrics, The International Biometric Society, vol. 78(1), pages 202-213, March.
    9. Haixia Liu & Zhongliang Zhou & Xiaojing Fan & Jiu Wang & Hongwei Sun & Chi Shen & Xiangming Zhai, 2020. "The Influence of Left-Behind Experience on College Students’ Mental Health: A Cross-Sectional Comparative Study," IJERPH, MDPI, vol. 17(5), pages 1-15, February.
    10. Cantone, Giulio Giacomo & Tomaselli, Venera, 2023. "Quasi-experimental network-based design for semantic analysis of small clusters of bi-polar online reviews," SocArXiv v7u3h, Center for Open Science.
    11. Mohammed Alyakoob & Mohammad S. Rahman & Zaiyan Wei, 2021. "Where You Live Matters: Local Bank Competition, Online Marketplace Lending, and Disparity in Borrower Benefits," Information Systems Research, INFORMS, vol. 32(4), pages 1390-1411, December.
    12. François Fall & Akim Almouksit, 2016. "The impact of formal financing on small informal enterprises in Comoros," Working Papers hal-01566389, HAL.
    13. Stefano Colonnello & Mariela Dal Borgo, 2024. "Raising Household Leverage: Evidence from Co-Financed Mortgages," Working Papers 2024: 01, Department of Economics, University of Venice "Ca' Foscari".
    14. Anna Harvey & Taylor Mattia, 2022. "Does money have a conservative bias? Estimating the causal impact of Citizens United on state legislative preferences," Public Choice, Springer, vol. 191(3), pages 417-441, June.
    15. Sergio Afcha & Jose García-Quevedo, 2016. "The impact of R&D subsidies on R&D employment composition," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 25(6), pages 955-975.
    16. Benjamin Bennett & Isil Erel & Léa H. Stern & Zexi Wang, 2020. "Paid Leave Pays Off: The Effects of Paid Family Leave on Firm Performance," NBER Working Papers 27788, National Bureau of Economic Research, Inc.
    17. Jing Wang & Gen Li & Kai-Lung Hui, 2022. "Monetary Incentives and Knowledge Spillover: Evidence from a Natural Experiment," Management Science, INFORMS, vol. 68(5), pages 3549-3572, May.
    18. Katie Meara & Francesco Pastore & Allan Webster, 2020. "The gender pay gap in the USA: a matching study," Journal of Population Economics, Springer;European Society for Population Economics, vol. 33(1), pages 271-305, January.
    19. Christophe Loussouarn & Carine Franc & Yann Videau & Julien Mousquès, 2021. "Can General Practitioners Be More Productive? The Impact of Teamwork and Cooperation with Nurses on GP Activities," Health Economics, John Wiley & Sons, Ltd., vol. 30(3), pages 680-698, March.
    20. Leduc, Elisabeth & Tojerow, Ilan, 2020. "Subsidizing Domestic Services as a Tool to Fight Unemployment: Effectiveness and Hidden Costs," IZA Discussion Papers 13544, Institute of Labor Economics (IZA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:reihed:v:62:y:2021:i:8:d:10.1007_s11162-021-09632-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.