IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2008.00602.html
   My bibliography  Save this paper

Design-Based Uncertainty for Quasi-Experiments

Author

Listed:
  • Ashesh Rambachan
  • Jonathan Roth

Abstract

Design-based frameworks of uncertainty are frequently used in settings where the treatment is (conditionally) randomly assigned. This paper develops a design-based framework suitable for analyzing quasi-experimental settings in the social sciences, in which the treatment assignment can be viewed as the realization of some stochastic process but there is concern about unobserved selection into treatment. In our framework, treatments are stochastic, but units may differ in their probabilities of receiving treatment, thereby allowing for rich forms of selection. We provide conditions under which the estimands of popular quasi-experimental estimators correspond to interpretable finite-population causal parameters. We characterize the biases and distortions to inference that arise when these conditions are violated. These results can be used to conduct sensitivity analyses when there are concerns about selection into treatment. Taken together, our results establish a rigorous foundation for quasi-experimental analyses that more closely aligns with the way empirical researchers discuss the variation in the data.

Suggested Citation

  • Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Jun 2025.
    • Handle: RePEc:arx:papers:2008.00602
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2008.00602
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. St'ephane Bonhomme & Martin Weidner, 2018. "Minimizing Sensitivity to Model Misspecification," Papers 1807.02161, arXiv.org, revised Oct 2021.
    2. Hyunseung Kang & Laura Peck & Luke Keele, 2018. "Inference for instrumental variables: a randomization inference approach," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1231-1254, October.
    3. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    4. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    5. Jiafeng Chen & Edward L. Glaeser & David Wessel, 2019. "The (Non-) Effect of Opportunity Zones on Housing Prices," NBER Working Papers 26587, National Bureau of Economic Research, Inc.
    6. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    7. Xinran Li & Peng Ding, 2017. "General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1759-1769, October.
    8. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    9. Douglas Staiger & James H. Stock, 1997. "Instrumental Variables Regression with Weak Instruments," Econometrica, Econometric Society, vol. 65(3), pages 557-586, May.
    10. Charles F. Manski & John V. Pepper, 2018. "How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions," The Review of Economics and Statistics, MIT Press, vol. 100(2), pages 232-244, May.
    11. Goodman, Joshua, 2008. "Who merits financial aid?: Massachusetts' Adams Scholarship," Journal of Public Economics, Elsevier, vol. 92(10-11), pages 2121-2131, October.
    12. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    13. Isaiah Andrews & Matthew Gentzkow & Jesse M. Shapiro, 2017. "Measuring the Sensitivity of Parameter Estimates to Estimation Moments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(4), pages 1553-1592.
    14. Vaart,A. W. van der, 2000. "Asymptotic Statistics," Cambridge Books, Cambridge University Press, number 9780521784504, Enero-Abr.
    15. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    16. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    17. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    18. Athey, Susan & Imbens, Guido W., 2022. "Design-based analysis in Difference-In-Differences settings with staggered adoption," Journal of Econometrics, Elsevier, vol. 226(1), pages 62-79.
    19. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ruonan Xu, 2023. "Difference-in-Differences with Interference," Papers 2306.12003, arXiv.org, revised Jan 2025.
    2. Jonathan Roth & Pedro H. C. Sant’Anna, 2023. "Efficient Estimation for Staggered Rollout Designs," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 1(4), pages 669-709.
    3. Ruonan Xu & Jeffrey M. Wooldridge, 2022. "A Design-Based Approach to Spatial Correlation," Papers 2211.14354, arXiv.org.
    4. Lin Winston & Dudoit Sandrine & Nolan Deborah & Speed Terence P., 2024. "From urn models to box models: Making Neyman's (1923) insights accessible," Journal of Causal Inference, De Gruyter, vol. 12(1), pages 1-12.
    5. Lea Bottmer & Guido Imbens & Jann Spiess & Merrill Warnick, 2021. "A Design-Based Perspective on Synthetic Control Methods," Papers 2101.09398, arXiv.org, revised Jul 2023.
    6. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    2. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    3. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    4. Pfeifer, Gregor & Stockburger, Mirjam, 2023. "The morning after: Prescription-free access to emergency contraceptive pills," Journal of Health Economics, Elsevier, vol. 91(C).
    5. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    6. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    7. Tamara Bischof & Boris Kaiser, 2021. "Who cares when you close down? The effects of primary care practice closures on patients," Health Economics, John Wiley & Sons, Ltd., vol. 30(9), pages 2004-2025, September.
    8. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    9. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    10. repec:osf:osfxxx:ac5ru_v1 is not listed on IDEAS
    11. Peter Z. Schochet, 2021. "Statistical Power for Estimating Treatment Effects Using Difference-in-Differences and Comparative Interrupted Time Series Designs with Variation in Treatment Timing," Papers 2102.06770, arXiv.org, revised Oct 2021.
    12. Jonathan Roth & Pedro H. C. Sant’Anna, 2023. "Efficient Estimation for Staggered Rollout Designs," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 1(4), pages 669-709.
    13. Baker, Andrew C. & Larcker, David F. & Wang, Charles C.Y., 2022. "How much should we trust staggered difference-in-differences estimates?," Journal of Financial Economics, Elsevier, vol. 144(2), pages 370-395.
    14. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    15. Luis Alvarez & Bruno Ferman, 2020. "Inference in Difference-in-Differences with Few Treated Units and Spatial Correlation," Papers 2006.16997, arXiv.org, revised Apr 2023.
    16. Wang, Lisha & Pan, Minjie & Qian, Xinlei & Lv, Kangjuan, 2025. "Do specialized courts matter? Environmental judiciary and corporate emissions in China," Energy Policy, Elsevier, vol. 199(C).
    17. Maclean, J. Catherine & Pichler, Stefan & Ziebarth, Nicolas R., 2020. "Mandated Sick Pay: Coverage, Utilization, and Welfare Effects," IZA Discussion Papers 13132, Institute of Labor Economics (IZA).
    18. Damian Clarke & Kathya Tapia-Schythe, 2021. "Implementing the panel event study," Stata Journal, StataCorp LLC, vol. 21(4), pages 853-884, December.
    19. Chy, Mahfuz & Kyung, Hoyoun, 2023. "The effect of bond market transparency on bank loan contracting," Journal of Accounting and Economics, Elsevier, vol. 75(2).
    20. Li, Tongxia & Ang, Tze Chuan ‘Chewie’ & Lu, Chun, 2023. "Employment protection and the provision of trade credit," Journal of Banking & Finance, Elsevier, vol. 155(C).
    21. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2008.00602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.