IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i2p179-208.html

Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting

Author

Listed:
  • Bruno Arpino
  • Silvia Bacci
  • Leonardo Grilli
  • Raffaele Guetto
  • Carla Rampichini

Abstract

We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.

Suggested Citation

  • Bruno Arpino & Silvia Bacci & Leonardo Grilli & Raffaele Guetto & Carla Rampichini, 2025. "Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting," Evaluation Review, , vol. 49(2), pages 179-208, April.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:179-208
    DOI: 10.1177/0193841X241246833
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241246833
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241246833?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hong, Guanglei & Raudenbush, Stephen W., 2006. "Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 901-910, September.
    2. Laura Forastiere & Edoardo M. Airoldi & Fabrizia Mealli, 2021. "Identification and Estimation of Treatment and Interference Effects in Observational Studies on Networks," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 901-918, April.
    3. Carrillo, Paul E. & Onofa, Mercedes & Ponce, Juan, 2010. "Information Technology and Student Achievement: Evidence from a Randomized Experiment in Ecuador," IDB Publications (Working Papers) 3094, Inter-American Development Bank.
    4. Arpino, Bruno & Mealli, Fabrizia, 2011. "The specification of the propensity score in multilevel observational studies," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1770-1780, April.
    5. Ding, Peng & Li, Fan, 2019. "A Bracketing Relationship between Difference-in-Differences and Lagged-Dependent-Variable Adjustment," Political Analysis, Cambridge University Press, vol. 27(4), pages 605-615, October.
    6. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    7. Bruno Arpino & Arnstein Aassve, 2013. "Estimating the causal effect of fertility on economic wellbeing: data requirements, identifying assumptions and estimation methods," Empirical Economics, Springer, vol. 44(1), pages 355-385, February.
    8. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, Enero-Abr.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mäkinen, Taneli & Li, Fan & Mercatanti, Andrea & Silvestrini, Andrea, 2022. "Causal analysis of central bank holdings of corporate bonds under interference," Economic Modelling, Elsevier, vol. 113(C).
    2. Nicolas Debarsy & Julie Le Gallo, 2025. "Identification of Spatial Spillovers: Do's and Don'ts," Journal of Economic Surveys, Wiley Blackwell, vol. 39(5), pages 2152-2173, December.
    3. repec:hal:journl:hal-04549691 is not listed on IDEAS
    4. Tadao Hoshino & Takahide Yanagi, 2024. "Causal Inference with Noncompliance and Unknown Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(548), pages 2869-2880, October.
    5. Wei, Wei & Young, Alex, 2025. "Beyond Russell reconstitution: A re-examination of methodologies for natural experiments," Journal of Corporate Finance, Elsevier, vol. 91(C).
    6. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Rocío Titiunik, 2019. "Regression Discontinuity Designs Using Covariates," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 442-451, July.
    7. Youmi Suk, 2024. "A Within-Group Approach to Ensemble Machine Learning Methods for Causal Inference in Multilevel Studies," Journal of Educational and Behavioral Statistics, , vol. 49(1), pages 61-91, February.
    8. Soojin Park & Peter M. Steiner & David Kaplan, 2018. "Identification and Sensitivity Analysis for Average Causal Mediation Effects with Time-Varying Treatments and Mediators: Investigating the Underlying Mechanisms of Kindergarten Retention Policy," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 298-320, June.
    9. Francesco Foglia, 2025. "The impact of EU Cohesion policy funds for Smart Specialisation in a lagging region: evidence from RDD approach," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 74(3), pages 1-23, September.
    10. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    11. Krishna Neupane & Prem Sapkota & Ujjwal Prajapati, 2026. "Beyond the Numbers: Causal Effects of Financial Report Sentiment on Bank Profitability," Papers 2602.17851, arXiv.org.
    12. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    13. Weicong Lyu & Jee-Seon Kim & Youmi Suk, 2023. "Estimating Heterogeneous Treatment Effects Within Latent Class Multilevel Models: A Bayesian Approach," Journal of Educational and Behavioral Statistics, , vol. 48(1), pages 3-36, February.
    14. Tadao Hoshino & Takahide Yanagi, 2023. "Randomization Test for the Specification of Interference Structure," Papers 2301.05580, arXiv.org, revised Dec 2023.
    15. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    16. Youmi Suk & Peter M. Steiner & Jee-Seon Kim & Hyunseung Kang, 2022. "Regression Discontinuity Designs With an Ordinal Running Variable: Evaluating the Effects of Extended Time Accommodations for English-Language Learners," Journal of Educational and Behavioral Statistics, , vol. 47(4), pages 459-484, August.
    17. Nadine Levratto & Aurelien Quignon, 2021. "Innovation Performance and the Signal Effect: Evidence from a European Program," Working Papers halshs-03466903, HAL.
    18. Emil Aas Stoltenberg, 2022. "Regression discontinuity design with right-censored survival data," Papers 2210.02548, arXiv.org.
    19. Pasquini, Ricardo A., 2021. "Effects of regulating the brokerage commission in the rental market: Evidence from Buenos Aires," Journal of Housing Economics, Elsevier, vol. 54(C).
    20. Kim, Gi-Soo & Paik, Myunghee Cho & Kim, Hongsoo, 2017. "Causal inference with observational data under cluster-specific non-ignorable assignment mechanism," Computational Statistics & Data Analysis, Elsevier, vol. 113(C), pages 88-99.
    21. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:179-208. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.