IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i22p4333-d977522.html
   My bibliography  Save this article

Evaluation of Educational Interventions Based on Average Treatment Effect: A Case Study

Author

Listed:
  • Jingyu Liang

    (School of Reliability and Systems Engineering, Beihang University, Beijing 100191, China)

  • Jie Liu

    (School of Reliability and Systems Engineering, Beihang University, Beijing 100191, China)

Abstract

Relative to randomized controlled trials (RCTs) with privacy and ethical concerns, observational studies are becoming dominant in education research. In an observational study, it is necessary and important to correctly evaluate the effects of different interventions (i.e., covariates) on student performance with observational data. However, these effects’ evaluation results are probably derived from biased estimations because the distributions of “control” and “treatment” student groups can hardly be equivalent to those in RCTs. Moreover, the collected covariates on possible educational interventions (i.e., treatments) may be confounded with student characteristics that are not included in the data. In this work, an estimation method based on the Rubin causal model (RCM) is proposed to calculate the average treatment effect (ATE) of different educational interventions. Specifically, with the selected covariates, the propensity score (i.e., the probability of treatment exposure conditional on covariates) is considered as a criterion to stratify the observational data into sub-classes with balanced covariate distributions between the control and treatment groups. Combined with Neyman’s estimation, the ATE of each sample is then obtained. We verify the effectiveness of this method with real observational data on student performance and its covariates.

Suggested Citation

  • Jingyu Liang & Jie Liu, 2022. "Evaluation of Educational Interventions Based on Average Treatment Effect: A Case Study," Mathematics, MDPI, vol. 10(22), pages 1-18, November.
    • Handle: RePEc:gam:jmathe:v:10:y:2022:i:22:p:4333-:d:977522
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/22/4333/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/10/22/4333/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Amit Sharma & Emre Kiciman, 2020. "DoWhy: An End-to-End Library for Causal Inference," Papers 2011.04216, arXiv.org.
    2. S Yang & P Ding, 2018. "Asymptotic inference of causal effects with observational studies trimmed by the estimated propensity scores," Biometrika, Biometrika Trust, vol. 105(2), pages 487-493.
    3. Jonathan M. Turk, 2019. "Estimating the Impact of Developmental Education on Associate Degree Completion: A Dose–Response Approach," Research in Higher Education, Springer;Association for Institutional Research, vol. 60(8), pages 1090-1112, December.
    4. Masserini, Lucio & Bini, Matilde, 2021. "Does joining social media groups help to reduce students’ dropout within the first university year?," Socio-Economic Planning Sciences, Elsevier, vol. 73(C).
    5. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    2. Xinwei Ma & Jingshen Wang, 2018. "Robust Inference Using Inverse Probability Weighting," Papers 1810.11397, arXiv.org, revised May 2019.
    3. Shu Yang & Yunshu Zhang, 2023. "Multiply robust matching estimators of average and quantile treatment effects," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 50(1), pages 235-265, March.
    4. Sven Resnjanskij & Jens Ruhose & Simon Wiederhold & Ludger Wößmann, 2021. "Mentoring Improves the Labor-Market Prospects of Highly Disadvantaged Adolescents," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 74(02), pages 31-38, February.
    5. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    6. Dimitris Bertsimas & Agni Orfanoudaki & Rory B. Weiner, 2020. "Personalized treatment for coronary artery disease patients: a machine learning approach," Health Care Management Science, Springer, vol. 23(4), pages 482-506, December.
    7. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    8. Clément de Chaisemartin & Luc Behaghel, 2020. "Estimating the Effect of Treatments Allocated by Randomized Waiting Lists," Econometrica, Econometric Society, vol. 88(4), pages 1453-1477, July.
    9. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    10. Bonesrønning, Hans & Finseraas, Henning & Hardoy, Ines & Iversen, Jon Marius Vaag & Nyhus, Ole Henning & Opheim, Vibeke & Salvanes, Kari Vea & Sandsør, Astrid Marie Jorde & Schøne, Pål, 2022. "Small-group instruction to improve student performance in mathematics in early grades: Results from a randomized field experiment," Journal of Public Economics, Elsevier, vol. 216(C).
    11. Peydró, José-Luis & Jiménez, Gabriel & Kenan, Huremovic & Moral-Benito, Enrique & Vega-Redondo, Fernando, 2020. "Production and financial networks in interplay: Crisis evidence from supplier-customer and credit registers," CEPR Discussion Papers 15277, C.E.P.R. Discussion Papers.
    12. Ruoxuan Xiong & Allison Koenecke & Michael Powell & Zhu Shen & Joshua T. Vogelstein & Susan Athey, 2021. "Federated Causal Inference in Heterogeneous Observational Data," Papers 2107.11732, arXiv.org, revised Apr 2023.
    13. Marie Bjørneby & Annette Alstadsæter & Kjetil Telle, 2018. "Collusive tax evasion by employers and employees. Evidence from a randomized fi eld experiment in Norway," Discussion Papers 891, Statistics Norway, Research Department.
    14. Konrad Menzel, 2021. "Structural Sieves," Papers 2112.01377, arXiv.org, revised Apr 2022.
    15. Susan Athey & Guido W. Imbens & Stefan Wager, 2018. "Approximate residual balancing: debiased inference of average treatment effects in high dimensions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(4), pages 597-623, September.
    16. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    17. Andrés Elberg & Pedro M. Gardete & Rosario Macera & Carlos Noton, 2019. "Dynamic effects of price promotions: field evidence, consumer search, and supply-side implications," Quantitative Marketing and Economics (QME), Springer, vol. 17(1), pages 1-58, March.
    18. Suresh de Mel & David McKenzie & Christopher Woodruff, 2019. "Labor Drops: Experimental Evidence on the Return to Additional Labor in Microenterprises," American Economic Journal: Applied Economics, American Economic Association, vol. 11(1), pages 202-235, January.
    19. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    20. Chenchuan (Mark) Li & Ulrich K. Müller, 2021. "Linear regression with many controls of limited explanatory power," Quantitative Economics, Econometric Society, vol. 12(2), pages 405-442, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:22:p:4333-:d:977522. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.