IDEAS home Printed from https://ideas.repec.org/a/aea/jecper/v31y2017i2p3-32.html
   My bibliography  Save this article

The State of Applied Econometrics: Causality and Policy Evaluation

Author

Listed:
  • Susan Athey
  • Guido W. Imbens

Abstract

In this paper, we discuss recent developments in econometrics that we view as important for empirical researchers working on policy evaluation questions. We focus on three main areas, in each case, highlighting recommendations for applied work. First, we discuss new research on identification strategies in program evaluation, with particular focus on synthetic control methods, regression discontinuity, external validity, and the causal interpretation of regression methods. Second, we discuss various forms of supplementary analyses, including placebo analyses as well as sensitivity and robustness analyses, intended to make the identification strategies more credible. Third, we discuss some implications of recent advances in machine learning methods for causal effects, including methods to adjust for differences between treated and control units in high-dimensional settings, and methods for identifying and estimating heterogenous treatment effects.

Suggested Citation

  • Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
  • Handle: RePEc:aea:jecper:v:31:y:2017:i:2:p:3-32
    Note: DOI: 10.1257/jep.31.2.3
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/articles?id=10.1257/jep.31.2.3
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=xSdnGd4KZxKsnohN5yAQ7_rzQLmHMs0o
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 128(2), pages 531-580.
    2. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    3. Charles F. Manski, 2013. "Response to the Review of ‘Public Policy in an Uncertain World’," Economic Journal, Royal Economic Society, vol. 0, pages 412-415, August.
    4. Giovanni Peri & Vasil Yasenov, 2015. "The Labor Market Effects of a Refugee Wave: Applying the Synthetic Control Method to the Mariel Boatlift," NBER Working Papers 21801, National Bureau of Economic Research, Inc.
    5. John List & Azeem Shaikh & Yang Xu, 2016. "Multiple Hypothesis Testing in Experimental Economics," Artefactual Field Experiments 00402, The Field Experiments Website.
    6. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    7. David Card, 1990. "The Impact of the Mariel Boatlift on the Miami Labor Market," ILR Review, Cornell University, ILR School, vol. 43(2), pages 245-257, January.
    8. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    9. Athey, Susan & Wager, Stefan, 2017. "Efficient Policy Learning," Research Papers 3506, Stanford University, Graduate School of Business.
    10. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    11. Joshua D. Angrist, 2004. "Treatment effect heterogeneity in theory and practice," Economic Journal, Royal Economic Society, vol. 114(494), pages 52-83, March.
    12. José R. Zubizarreta, 2015. "Stable Weights that Balance Covariates for Estimation With Incomplete Outcome Data," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 910-922, September.
    13. repec:ags:stataj:267111 is not listed on IDEAS
    14. Bryan S. Graham & Cristine Campos de Xavier Pinto & Daniel Egel, 2016. "Efficient Estimation of Data Combination Models by the Method of Auxiliary-to-Study Tilting (AST)," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(2), pages 288-301, April.
    15. David Card & David S. Lee & Zhuan Pei & Andrea Weber, 2015. "Inference on Causal Effects in a Generalized Regression Kink Design," Econometrica, Econometric Society, vol. 83, pages 2453-2483, November.
    16. Taisuke Otsu & Ke-Li Xu & Yukitoshi Matsushita, 2013. "Estimation and Inference of Discontinuity in Density," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(4), pages 507-524, October.
    17. Scott E. Carrell & Bruce I. Sacerdote & James E. West, 2013. "From Natural Variation to Optimal Policy? The Importance of Endogenous Peer Group Formation," Econometrica, Econometric Society, vol. 81(3), pages 855-882, May.
    18. James J. Heckman & Vytlacil, Edward J., 2007. "Econometric Evaluation of Social Programs, Part II: Using the Marginal Treatment Effect to Organize Alternative Econometric Estimators to Evaluate Social Programs, and to Forecast their Effects in New," Handbook of Econometrics,in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 71 Elsevier.
    19. Jon Kleinberg & Jens Ludwig & Sendhil Mullainathan & Ziad Obermeyer, 2015. "Prediction Policy Problems," American Economic Review, American Economic Association, vol. 105(5), pages 491-495, May.
    20. Unknown, 2015. "Book Reviews," Indian Journal of Agricultural Economics, Indian Society of Agricultural Economics, vol. 70(1), pages 1-7.
    21. Sandra E. Black, 1999. "Do Better Schools Matter? Parental Valuation of Elementary Education," The Quarterly Journal of Economics, Oxford University Press, vol. 114(2), pages 577-599.
    22. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    23. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney K. Newey, 2016. "Double machine learning for treatment and causal parameters," CeMMAP working papers CWP49/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    24. Helena Skyt Nielsen & Torben Sørensen & Christopher Taber, 2010. "Estimating the Effect of Student Aid on College Enrollment: Evidence from a Government Grant Policy Reform," NBER Chapters,in: Income Taxation, Trans-Atlantic Public Economics Seminar (TAPES), pages 185-215 National Bureau of Economic Research, Inc.
    25. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
    26. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    27. Alexandre Belloni & Victor Chernozhukov & Ivan Fernandez-Val & Christian Hansen, 2013. "Program evaluation with high-dimensional data," CeMMAP working papers CWP77/13, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    28. Nicholas Christakis & James Fowler & Guido Imbens & Karthik Kalyanaraman, 2010. "An empirical model for strategic network formation," CeMMAP working papers CWP16/10, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    29. Angelo Mele, 2010. "A structural model of segregation in social networks," CeMMAP working papers CWP32/10, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    30. repec:ags:stataj:116250 is not listed on IDEAS
    31. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    32. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 933-959.
    33. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    34. Guido W. Imbens & Donald B. Rubin & Bruce I. Sacerdote, 2001. "Estimating the Effect of Unearned Income on Labor Earnings, Savings, and Consumption: Evidence from a Survey of Lottery Players," American Economic Review, American Economic Association, vol. 91(4), pages 778-794, September.
    35. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    36. Amanda Kowalski, 2016. "Doing more when you're running LATE: Applying marginal treatment effect methods to examine treatment effect heterogeneity in experiments," Artefactual Field Experiments 00560, The Field Experiments Website.
    37. Calonico, Sebastian & Cattaneo, Matias D. & Titiunik, Rocío, 2014. "Robust data-driven inference in the regression-discontinuity design," Stata Journal, StataCorp LP, vol. 14(4).
    38. repec:aea:aecrev:v:107:y:2017:i:5:p:278-81 is not listed on IDEAS
    39. repec:hrv:faseco:34728615 is not listed on IDEAS
    40. repec:ucp:jpolec:doi:10.1086/692712 is not listed on IDEAS
    41. James J. Heckman & Vytlacil, Edward J., 2007. "Econometric Evaluation of Social Programs, Part I: Causal Models, Structural Models and Econometric Policy Evaluation," Handbook of Econometrics,in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 70 Elsevier.
    42. Andrew Gelman & Guido Imbens, 2014. "Why High-order Polynomials Should not be Used in Regression Discontinuity Designs," NBER Working Papers 20405, National Bureau of Economic Research, Inc.
    43. Paul Goldsmith-Pinkham & Guido W. Imbens, 2013. "Social Networks and the Identification of Peer Effects," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(3), pages 253-264, July.
    44. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    45. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
    46. Hausman, Jerry, 2015. "Specification tests in econometrics," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 38(2), pages 112-134.
    47. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    48. Gary Chamberlain & Guido Imbens, 2004. "Random Effects Estimators with many Instrumental Variables," Econometrica, Econometric Society, vol. 72(1), pages 295-306, January.
    49. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    50. Elie Tamer, 2010. "Partial Identification in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 167-195, September.
    51. Manski, Charles F., 2013. "Public Policy in an Uncertain World: Analysis and Decisions," Economics Books, Harvard University Press, number 9780674066892, December.
    52. Yingying Dong & Arthur Lewbel, 2015. "Identifying the Effect of Changing the Policy Threshold in Regression Discontinuity Models," The Review of Economics and Statistics, MIT Press, vol. 97(5), pages 1081-1092, December.
    53. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    54. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    55. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    56. N/A, 2015. "Books for review," Work, Employment & Society, British Sociological Association, vol. 29(6), pages 1046-1046, December.
    57. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, Oxford University Press, vol. 130(3), pages 1117-1165.
    58. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    59. n/a, 2015. "Book Reviews," The Energy Journal, International Association for Energy Economics, vol. 0(Number 4).
    60. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    61. Marinho Bertanha & Guido W. Imbens, 2014. "External Validity in Fuzzy Regression Discontinuity Designs," NBER Working Papers 20773, National Bureau of Economic Research, Inc.
    62. Unknown, 2015. "Book Review," Indian Journal of Agricultural Economics, Indian Society of Agricultural Economics, vol. 70(2), pages 1-4.
    63. Bryan S. Graham & Cristine Campos De Xavier Pinto & Daniel Egel, 2012. "Inverse Probability Tilting for Moment Condition Models with Missing Data," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 1053-1079.
    64. Vaart,A. W. van der, 2000. "Asymptotic Statistics," Cambridge Books, Cambridge University Press, number 9780521784504.
    65. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    66. Wilbert van der Klaauw, 2008. "Regression-Discontinuity Analysis: A Survey of Recent Developments in Economics," LABOUR, CEIS, vol. 22(2), pages 219-245, June.
    67. Athey, Susan & Eckles, Dean & Imbens, Guido W., 2015. "Exact P-Values for Network Interference," Research Papers 3351, Stanford University, Graduate School of Business.
    68. Joshua Angrist & Ivan Fernandez-Val, 2010. "ExtrapoLATE-ing: External Validity and Overidentification in the LATE Framework," NBER Working Papers 16566, National Bureau of Economic Research, Inc.
    69. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    70. n/a, 2015. "Book Reviews," Economics of Energy & Environmental Policy, International Association for Energy Economics, vol. 0(Number 1).
    71. Matsudaira, Jordan D., 2008. "Mandatory summer school and student achievement," Journal of Econometrics, Elsevier, vol. 142(2), pages 829-850, February.
    72. Wilbert van der Klaauw, 2002. "Estimating the Effect of Financial Aid Offers on College Enrollment: A Regression-Discontinuity Approach," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 43(4), pages 1249-1287, November.
    73. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    74. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    75. repec:bla:biomet:v:72:y:2016:i:4:p:1055-1065 is not listed on IDEAS
    76. Abhijit Banerjee & Sylvain Chassang & Erik Snowberg, 2016. "Decision Theoretic Approaches to Experiment Design and External Validity," NBER Working Papers 22167, National Bureau of Economic Research, Inc.
    77. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    78. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2008. "Using Selection on Observed Variables to Assess Bias from Unobservables When Evaluating Swan-Ganz Catheterization," American Economic Review, American Economic Association, vol. 98(2), pages 345-350, May.
    79. Abadie, Alberto & Drukker, David M. & Herr, Jane Leber & Imbens, Guido W., 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 1-22.
    80. Susan Athey & Guido Imbens & Thai Pham & Stefan Wager, 2017. "Estimating Average Treatment Effects: Supplementary Analyses and Remaining Challenges," American Economic Review, American Economic Association, vol. 107(5), pages 278-281, May.
    81. Susan Athey & Guido Imbens, 2015. "A Measure of Robustness to Misspecification," American Economic Review, American Economic Association, vol. 105(5), pages 476-480, May.
    82. Timothy G. Conley & Christian B. Hansen & Peter E. Rossi, 2012. "Plausibly Exogenous," The Review of Economics and Statistics, MIT Press, vol. 94(1), pages 260-272, February.
    83. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 114(2), pages 533-575.
    84. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    85. Joshua D. Angrist & Miikka Rokkanen, 2015. "Wanna Get Away? Regression Discontinuity Estimation of Exam School Effects Away From the Cutoff," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1331-1344, December.
    86. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    87. Hahn, Jinyong & Todd, Petra & Van der Klaauw, Wilbert, 2001. "Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design," Econometrica, Econometric Society, vol. 69(1), pages 201-209, January.
    Full references (including those not matched with items on IDEAS)

    More about this item

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C50 - Mathematical and Quantitative Methods - - Econometric Modeling - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:31:y:2017:i:2:p:3-32. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Michael P. Albert). General contact details of provider: http://edirc.repec.org/data/aeaaaea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.