IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2002.00202.html
   My bibliography  Save this paper

Natural Experiments

Author

Listed:
  • Rocio Titiunik

Abstract

The term natural experiment is used inconsistently. In one interpretation, it refers to an experiment where a treatment is randomly assigned by someone other than the researcher. In another interpretation, it refers to a study in which there is no controlled random assignment, but treatment is assigned by some external factor in a way that loosely resembles a randomized experiment---often described as an "as if random" assignment. In yet another interpretation, it refers to any non-randomized study that compares a treatment to a control group, without any specific requirements on how the treatment is assigned. I introduce an alternative definition that seeks to clarify the integral features of natural experiments and at the same time distinguish them from randomized controlled experiments. I define a natural experiment as a research study where the treatment assignment mechanism (i) is neither designed nor implemented by the researcher, (ii) is unknown to the researcher, and (iii) is probabilistic by virtue of depending on an external factor. The main message of this definition is that the difference between a randomized controlled experiment and a natural experiment is not a matter of degree, but of essence, and thus conceptualizing a natural experiment as a research design akin to a randomized experiment is neither rigorous nor a useful guide to empirical analysis. Using my alternative definition, I discuss how a natural experiment differs from a traditional observational study, and offer practical recommendations for researchers who wish to use natural experiments to study causal effects.

Suggested Citation

  • Rocio Titiunik, 2020. "Natural Experiments," Papers 2002.00202, arXiv.org.
    • Handle: RePEc:arx:papers:2002.00202
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2002.00202
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erikson, Robert S. & Stoker, Laura, 2011. "Caught in the Draft: The Effects of Vietnam Draft Lottery Status on Political Attitudes," American Political Science Review, Cambridge University Press, vol. 105(2), pages 221-237, May.
    2. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    3. Joshua D. Angrist & Alan B. Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 69-85, Fall.
    4. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    5. Titiunik, Rocío, 2016. "Drawing Your Senator from a Jar:Term Length and Legislative Behavior," Political Science Research and Methods, Cambridge University Press, vol. 4(2), pages 293-316, May.
    6. Angrist, Joshua D, 1990. "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records: Errata," American Economic Review, American Economic Association, vol. 80(5), pages 1284-1286, December.
    7. Eric D. Gould & Victor Lavy & M. Daniele Paserman, 2004. "Immigrating to Opportunity: Estimating the Effect of School Quality Using a Natural Experiment on Ethiopians in Israel," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(2), pages 489-526.
    8. Sekhon, Jasjeet S. & Titiunik, Rocío, 2012. "When Natural Experiments Are Neither Natural nor Experiments," American Political Science Review, Cambridge University Press, vol. 106(1), pages 35-57, February.
    9. Bhavnani, Rikhil R., 2009. "Do Electoral Quotas Work after They Are Withdrawn? Evidence from a Natural Experiment in India," American Political Science Review, Cambridge University Press, vol. 103(1), pages 23-35, February.
    10. Matias D. Cattaneo & Nicolas Idrobo & Rocio Titiunik, 2019. "A Practical Introduction to Regression Discontinuity Designs: Foundations," Papers 1911.09511, arXiv.org.
    11. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    12. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    13. Dunning,Thad, 2012. "Natural Experiments in the Social Sciences," Cambridge Books, Cambridge University Press, number 9781107017665, November.
    14. Dunning,Thad, 2012. "Natural Experiments in the Social Sciences," Cambridge Books, Cambridge University Press, number 9781107698000, November.
    15. Jared Carbone & Daniel Hallstrom & V. Smith, 2006. "Can Natural Experiments Measure Behavioral Responses to Environmental Risks?," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 33(3), pages 273-297, March.
    16. Burt S. Barnow & Matias D. Cattaneo & Rocío Titiunik & Gonzalo Vazquez‐Bare, 2017. "Comparing Inference Approaches for RD Designs: A Reexamination of the Effect of Head Start on Child Mortality," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 36(3), pages 643-681, June.
    17. Joshua Angrist & Alan Krueger, 2001. "Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments," Working Papers 834, Princeton University, Department of Economics, Industrial Relations Section..
    18. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, January.
    19. Edward Miguel & Shanker Satyanath & Ernest Sergenti, 2004. "Economic Shocks and Civil Conflict: An Instrumental Variables Approach," Journal of Political Economy, University of Chicago Press, vol. 112(4), pages 725-753, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Baldwin, Kate & Bhavnani, Rikhil R., 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series 024, World Institute for Development Economic Research (UNU-WIDER).
    2. Kate Baldwin & Rikhil R. Bhavnani, 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series wp-2013-024, World Institute for Development Economic Research (UNU-WIDER).
    3. Nobel Prize Committee, 2021. "Answering causal questions using observational data," Nobel Prize in Economics documents 2021-2, Nobel Prize Committee.
    4. Anustubh Agnihotri & Rahul Verma, 2016. "Design-based Approach in Social Science Research," Studies in Indian Politics, , vol. 4(2), pages 241-248, December.
    5. Baldwin Kate & Bhavnani Rikhil R., 2015. "Ancillary Studies of Experiments: Opportunities and Challenges," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 113-146, June.
    6. Guido W. Imbens, 2022. "Causality in Econometrics: Choice vs Chance," Econometrica, Econometric Society, vol. 90(6), pages 2541-2566, November.
    7. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    8. Öberg, Stefan, 2021. "Treatment for natural experiments: How to improve causal estimates using conceptual definitions and substantive interpretations," SocArXiv pkyue, Center for Open Science.
    9. Kugler Franziska & Schwerdt Guido & Wößmann Ludger, 2014. "Ökonometrische Methoden zur Evaluierung kausaler Effekte der Wirtschaftspolitik," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 15(2), pages 105-132, June.
    10. David A. Bateman & Dawn Langan Teele, 2020. "A developmental approach to historical causal inference," Public Choice, Springer, vol. 185(3), pages 253-279, December.
    11. Öberg, Stefan, 2018. "Instrumental variables based on twin births are by definition not valid (v.3.0)," SocArXiv zux9s, Center for Open Science.
    12. Susan Athey & Raj Chetty & Guido Imbens, 2020. "Using Experiments to Correct for Selection in Observational Studies," Papers 2006.09676, arXiv.org, revised May 2025.
    13. Haoge Chang & Joel Middleton & P. M. Aronow, 2021. "Exact Bias Correction for Linear Adjustment of Randomized Controlled Trials," Papers 2110.08425, arXiv.org, revised Oct 2021.
    14. Gregory J. Wawro & Ira Katznelson, 2020. "American political development and new challenges of causal inference," Public Choice, Springer, vol. 185(3), pages 299-314, December.
    15. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    16. Florence Kondylis & Valerie Mueller, 2014. "Economic consequences of conflict and environmental displacement," Chapters, in: Robert E.B. Lucas (ed.), International Handbook on Migration and Economic Development, chapter 14, pages 388-424, Edward Elgar Publishing.
    17. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    18. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    19. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    20. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2002.00202. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.