IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i2p179-208.html

Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting

Author

Listed:
  • Bruno Arpino
  • Silvia Bacci
  • Leonardo Grilli
  • Raffaele Guetto
  • Carla Rampichini

Abstract

We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.

Suggested Citation

  • Bruno Arpino & Silvia Bacci & Leonardo Grilli & Raffaele Guetto & Carla Rampichini, 2025. "Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting," Evaluation Review, , vol. 49(2), pages 179-208, April.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:179-208
    DOI: 10.1177/0193841X241246833
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241246833
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241246833?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bruno Arpino & Arnstein Aassve, 2013. "Estimating the causal effect of fertility on economic wellbeing: data requirements, identifying assumptions and estimation methods," Empirical Economics, Springer, vol. 44(1), pages 355-385, February.
    2. Hong, Guanglei & Raudenbush, Stephen W., 2006. "Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 901-910, September.
    3. Laura Forastiere & Edoardo M. Airoldi & Fabrizia Mealli, 2021. "Identification and Estimation of Treatment and Interference Effects in Observational Studies on Networks," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(534), pages 901-918, April.
    4. Carrillo, Paul E. & Onofa, Mercedes & Ponce, Juan, 2010. "Information Technology and Student Achievement: Evidence from a Randomized Experiment in Ecuador," IDB Publications (Working Papers) 3094, Inter-American Development Bank.
    5. Arpino, Bruno & Mealli, Fabrizia, 2011. "The specification of the propensity score in multilevel observational studies," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1770-1780, April.
    6. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, Enero-Abr.
    7. Ding, Peng & Li, Fan, 2019. "A Bracketing Relationship between Difference-in-Differences and Lagged-Dependent-Variable Adjustment," Political Analysis, Cambridge University Press, vol. 27(4), pages 605-615, October.
    8. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mäkinen, Taneli & Li, Fan & Mercatanti, Andrea & Silvestrini, Andrea, 2022. "Causal analysis of central bank holdings of corporate bonds under interference," Economic Modelling, Elsevier, vol. 113(C).
    2. Nicolas Debarsy & Julie Le Gallo, 2025. "Identification of Spatial Spillovers: Do's and Don'ts," Journal of Economic Surveys, Wiley Blackwell, vol. 39(5), pages 2152-2173, December.
    3. repec:hal:journl:hal-04549691 is not listed on IDEAS
    4. Tadao Hoshino & Takahide Yanagi, 2024. "Causal Inference with Noncompliance and Unknown Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(548), pages 2869-2880, October.
    5. Hao, Shiming, 2021. "True structure change, spurious treatment effect? A novel approach to disentangle treatment effects from structure changes," MPRA Paper 108679, University Library of Munich, Germany.
    6. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," NBER Working Papers 26080, National Bureau of Economic Research, Inc.
    7. Brantly Callaway & Derek Dyal & Pedro H. C. Sant'Anna & Emmanuel S. Tsyawo, 2025. "Beyond Parallel Trends: An Identification-Strategy-Robust Approach to Causal Inference with Panel Data," Papers 2511.21977, arXiv.org.
    8. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    9. Wei, Wei & Young, Alex, 2025. "Beyond Russell reconstitution: A re-examination of methodologies for natural experiments," Journal of Corporate Finance, Elsevier, vol. 91(C).
    10. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    11. Sebastian Calonico & Matias D. Cattaneo & Max H. Farrell & Rocío Titiunik, 2019. "Regression Discontinuity Designs Using Covariates," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 442-451, July.
    12. Youmi Suk, 2024. "A Within-Group Approach to Ensemble Machine Learning Methods for Causal Inference in Multilevel Studies," Journal of Educational and Behavioral Statistics, , vol. 49(1), pages 61-91, February.
    13. Jens Forderer & Gordon Burtch, 2025. "Estimating Career Benefits from Online Community Leadership: Evidence from Stack Exchange Moderators," Management Science, INFORMS, vol. 71(3), pages 2443-2466, March.
    14. Soojin Park & Peter M. Steiner & David Kaplan, 2018. "Identification and Sensitivity Analysis for Average Causal Mediation Effects with Time-Varying Treatments and Mediators: Investigating the Underlying Mechanisms of Kindergarten Retention Policy," Psychometrika, Springer;The Psychometric Society, vol. 83(2), pages 298-320, June.
    15. Francesco Foglia, 2025. "The impact of EU Cohesion policy funds for Smart Specialisation in a lagging region: evidence from RDD approach," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 74(3), pages 1-23, September.
    16. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    17. Aditya Ghosh & Stefan Wager, 2025. "Non-parametric Causal Inference in Dynamic Thresholding Designs," Papers 2512.15244, arXiv.org, revised May 2026.
    18. Krishna Neupane & Prem Sapkota & Ujjwal Prajapati, 2026. "Beyond the Numbers: Causal Effects of Financial Report Sentiment on Bank Profitability," Papers 2602.17851, arXiv.org.
    19. Mary Ying-Fang Wang & Paul Tuss & Lihong Qi, 2019. "Augmented Weighted Estimators Dealing with Practical Positivity Violation to Causal inferences in a Random Coefficient Model," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 447-467, June.
    20. Fabrizia Mealli & Javier Viviens, 2025. "Difference-in-Differences in the Presence of Unknown Interference," Papers 2512.21176, arXiv.org, revised Feb 2026.
    21. Cavalletti, Barbara & Corsi, Matteo & Persico, Luca & di Bella, Enrico, 2021. "Public university orientation for high-school students. A quasi-experimental assessment of the efficiency gains from nudging better career choices," Socio-Economic Planning Sciences, Elsevier, vol. 73(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:2:p:179-208. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.