IDEAS home Printed from https://ideas.repec.org/p/cte/werepe/50593.html

Big Bias from Small Treatment Heterogeneity: When Controlling for Selection Backfires

Author

Listed:
  • Carro, Jesús M.
  • Von Lampe, Gregor

Abstract

Applied work routinely estimates treatment effects by linear regression of theoutcome on a treatment indicator and controls, with no interactions between them. We show that, even when selection is entirely on observables and the linear model is correctly specified, minimal unmodeled treatment effect heterogeneity can produce large biases for the average treatment effect on the treated. We decompose the asymptotic bias, control by control, into the product of an unobservable heterogeneity component and an observable amplifier, estimable without outcome data, that grows without bound as the controls better explain treatment. Covariates irrelevant for the outcome are not innocuous: they can generate large biases while their spuriously significant estimates make them self-validating. We propose breakdown diagnostics: the minimal correlated heterogeneity that overturns an estimate. In difference-in-differences, group-specific trends, a standard robustness check, absorb the dynamics of the treatment effect, producing a bias that accumulates with the length of the panel. Applications to 401(k) eligibility and unilateral divorce laws illustrate.

Suggested Citation

  • Carro, Jesús M. & Von Lampe, Gregor, 2026. "Big Bias from Small Treatment Heterogeneity: When Controlling for Selection Backfires," UC3M Working papers. Economics 50593, Universidad Carlos III de Madrid. Departamento de Economía.
  • Handle: RePEc:cte:werepe:50593
    as

    Download full text from publisher

    File URL: https://e-archivo.uc3m.es/rest/api/core/bitstreams/bc1503b6-d843-4f15-8fe0-e80f05dafc41/content
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jonathan Meer & Jeremy West, 2016. "Effects of the Minimum Wage on Employment Dynamics," Journal of Human Resources, University of Wisconsin Press, vol. 51(2), pages 500-522.
    2. Paul Goldsmith-Pinkham & Peter Hull & Michal Kolesár, 2024. "Contamination Bias in Linear Regressions," American Economic Review, American Economic Association, vol. 114(12), pages 4015-4051, December.
    3. Soonwoo Kwon & Liyang Sun, 2025. "Estimating Treatment Effects Under Bounded Heterogeneity," Papers 2510.05454, arXiv.org, revised Aug 2026.
    4. Emily Oster, 2019. "Unobservable Selection and Coefficient Stability: Theory and Evidence," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 187-204, April.
    5. Patrick Kline, 2011. "Oaxaca-Blinder as a Reweighting Estimator," American Economic Review, American Economic Association, vol. 101(3), pages 532-537, May.
    6. Ambarish Chattopadhyay & José R Zubizarreta, 2023. "On the implied weights of linear regression for causal inference," Biometrika, Biometrika Trust, vol. 110(3), pages 615-629.
    7. Leora Friedberg, 1998. "Did Unilateral Divorce Raise Divorce Rates? Evidence from Panel Data," NBER Working Papers 6398, National Bureau of Economic Research, Inc.
    8. Clément de Chaisemartin & Xavier D’Haultfœuille, 2023. "Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey," The Econometrics Journal, Royal Economic Society, vol. 26(3), pages 1-30.
    9. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    10. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    11. Matthew A. Masten & Alexandre Poirier, 2020. "Inference on breakdown frontiers," Quantitative Economics, Econometric Society, vol. 11(1), pages 41-111, January.
    12. Victor Chernozhukov & Christian Hansen, 2004. "The Effects of 401(K) Participation on the Wealth Distribution: An Instrumental Quantile Regression Analysis," The Review of Economics and Statistics, MIT Press, vol. 86(3), pages 735-751, August.
    13. Christine Blandhol & John Bonney & Magne Mogstad & Alexander Torgovitsky, 2022. "When is TSLS Actually LATE?," NBER Working Papers 29709, National Bureau of Economic Research, Inc.
    14. Poterba, James M. & Venti, Steven F. & Wise, David A., 1995. "Do 401(k) contributions crowd out other personal saving?," Journal of Public Economics, Elsevier, vol. 58(1), pages 1-32, September.
    15. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    16. Benjamin, Daniel J., 2003. "Does 401(k) eligibility increase saving?: Evidence from propensity score subclassification," Journal of Public Economics, Elsevier, vol. 87(5-6), pages 1259-1290, May.
    17. Justin Wolfers, 2006. "Did Unilateral Divorce Laws Raise Divorce Rates? A Reconciliation and New Results," American Economic Review, American Economic Association, vol. 96(5), pages 1802-1820, December.
    18. Joshua D. Angrist, 1998. "Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants," Econometrica, Econometric Society, vol. 66(2), pages 249-288, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Carballido Cerqueiro, María & Carro, Jesús M. & Siotis, Georgios, 2026. "Multimarket contact and the Southwest effect: twenty-five years of evolving rivalry in the US airline industry," UC3M Working papers. Economics 50949, Universidad Carlos III de Madrid. Departamento de Economía.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mogstad, Magne & Torgovitsky, Alexander, 2024. "Instrumental variables with unobserved heterogeneity in treatment effects," Handbook of Labor Economics,, Elsevier.
    2. Soonwoo Kwon & Liyang Sun, 2025. "Estimating Treatment Effects Under Bounded Heterogeneity," Papers 2510.05454, arXiv.org, revised Aug 2026.
    3. Angelini, Viola & Bertoni, Marco & Stella, Luca & Weiss, Christoph T., 2019. "The ant or the grasshopper? The long-term consequences of Unilateral Divorce Laws on savings of European households," European Economic Review, Elsevier, vol. 119(C), pages 97-113.
    4. Diego F. Salazar & Noah Lim & Ingrid Koch, 2026. "Job Security, Gender, and Sales Performance: Evidence from a Retail Sales Context," Marketing Science, INFORMS, vol. 45(1), pages 80-107, January.
    5. Michael C. Knaus, 2024. "Treatment Effect Estimators as Weighted Outcomes," Papers 2411.11559, arXiv.org, revised Dec 2024.
    6. Andrea Cintolesi & Sauro Mocetti & Giacomo Roma, 2024. "Productivity and entry regulation: evidence from the universe of firms," Temi di discussione (Economic working papers) 1455, Bank of Italy, Economic Research and International Relations Area.
    7. Alloush, Mo & Bloem, Jeffrey R., 2022. "Neighborhood violence, poverty, and psychological well-being," Journal of Development Economics, Elsevier, vol. 154(C).
    8. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, IZA Network @ LISER.
    9. Gruber, Jonathan & Lin, Mengyun & Yi, Junjian, 2023. "The largest insurance program in history: Saving one million lives per year in China," Journal of Public Economics, Elsevier, vol. 226(C).
    10. Yu, Ping & Phillips, Peter C.B., 2018. "Threshold regression with endogeneity," Journal of Econometrics, Elsevier, vol. 203(1), pages 50-68.
    11. Liu, Lisa Yao & Tong, Lijing, 2026. "The spillover effect of public firm audit regulation on private firm auditing: Evidence from common partners," Journal of Accounting and Economics, Elsevier, vol. 81(3).
    12. Ober-Reynolds, Daniel, 2026. "Robustness to missing data: breakdown point analysis," Journal of Econometrics, Elsevier, vol. 253(C).
    13. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2020. "Testing Unconfoundedness Assumption Using Auxiliary Variables," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202004, University of Kansas, Department of Economics, revised Feb 2020.
    14. Lin, Zhexiao & Han, Fang, 2025. "On regression-adjusted imputation estimators of average treatment effects," Journal of Econometrics, Elsevier, vol. 251(C).
    15. Pietro Emilio Spini, 2021. "A Simple Measure of Robustness for External Validity under Covariate Shifts," Papers 2112.09259, arXiv.org, revised May 2026.
    16. Joakim A. Weill & Matthieu Stigler & Olivier Deschenes & Michael R. Springborn, 2025. "Researchers' degrees of flexibility: Revisiting COVID‐19 policy evaluations," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 441-462, April.
    17. Tymon S{l}oczy'nski, 2018. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," Papers 1810.01576, arXiv.org, revised May 2020.
    18. Corneo, Giacomo & Keese, Matthias & Schröder, Carsten, 2010. "The effect of saving subsidies on household saving: Evidence from Germany," Discussion Papers 2010/3, Free University Berlin, School of Business & Economics.
    19. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    20. Stefanie Thönnes, 2019. "Ex-post moral hazard in the health insurance market: empirical evidence from German data," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(9), pages 1317-1333, December.

    More about this item

    Keywords

    ;

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cte:werepe:50593. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ana Poveda (email available below). General contact details of provider: http://www.eco.uc3m.es/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.