IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this article or follow this journal

The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics

  • Joshua D. Angrist
  • J�rn-Steffen Pischke

Since Edward Leamer's memorable 1983 paper, "Let's Take the Con out of Econometrics," empirical microeconomics has experienced a credibility revolution. While Leamer's suggested remedy, sensitivity analysis, has played a role in this, we argue that the primary engine driving improvement has been a focus on the quality of empirical research designs. The advantages of a good research design are perhaps most easily apparent in research using random assignment. We begin with an overview of Leamer's 1983 critique and his proposed remedies. We then turn to the key factors we see contributing to improved empirical work, including the availability of more and better data, along with advances in theoretical econometric understanding, but especially the fact that research design has moved front and center in much of empirical micro. We offer a brief digression into macroeconomics and industrial organization, where progress -- by our lights -- is less dramatic, although there is work in both fields that we find encouraging. Finally, we discuss the view that the design pendulum has swung too far. Critics of design-driven studies argue that in pursuit of clean and credible research designs, researchers seek good answers instead of good questions. We briefly respond to this concern, which worries us little.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.aeaweb.org/articles.php?doi=10.1257/jep.24.2.3
Download Restriction: no

Article provided by American Economic Association in its journal Journal of Economic Perspectives.

Volume (Year): 24 (2010)
Issue (Month): 2 (Spring)
Pages: 3-30

as
in new window

Handle: RePEc:aea:jecper:v:24:y:2010:i:2:p:3-30
Note: DOI: 10.1257/jep.24.2.3
Contact details of provider: Web page: https://www.aeaweb.org/jep/Email:


More information through EDIRC

Order Information: Web: https://www.aeaweb.org/subscribe.html

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Joshua D. Angrist & Stacey H. Chen, 2009. "Long-Term Economic Consequences of Vietnam-Era Conscription: Schooling, Experience and Earnings," Royal Holloway, University of London: Discussion Papers in Economics 09/02, Department of Economics, Royal Holloway University of London.
  2. N. G. Mankiw., 2009. "The Macroeconomist as Scientist and Engineer," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 5.
  3. Bruce D. Meyer, 1994. "Natural and Quasi- Experiments in Economics," NBER Technical Working Papers 0170, National Bureau of Economic Research, Inc.
  4. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
  5. Raghuram G. Rajan & Arvind Subramanian, 2005. "Aid and Growth: What Does the Cross-Country Evidence Really Show?," NBER Working Papers 11513, National Bureau of Economic Research, Inc.
  6. Ehrlich, Isaac, 1977. "The Deterrent Effect of Capital Punishment: Reply," American Economic Review, American Economic Association, vol. 67(3), pages 452-58, June.
  7. Card, David & Krueger, Alan B, 1992. "School Quality and Black-White Relative Earnings: A Direct Assessment," The Quarterly Journal of Economics, MIT Press, vol. 107(1), pages 151-200, February.
  8. Christina D. Romer & David H. Romer, 2007. "The Macroeconomic Effects of Tax Changes: Estimates Based on a New Measure of Fiscal Shocks," NBER Working Papers 13264, National Bureau of Economic Research, Inc.
  9. Jeffrey R. Kling & Jeffrey B. Liebman & Lawrence F. Katz, 2005. "Experimental Analysis of Neighborhood Effects," NBER Working Papers 11577, National Bureau of Economic Research, Inc.
  10. Imbens, G. & Angrist, J.D., 1992. "Average Causal Response with Variable Treatment Intensity," Harvard Institute of Economic Research Working Papers 1611, Harvard - Institute of Economic Research.
  11. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
  12. Ramey, Valerie A. & Shapiro, Matthew D., 1998. "Costly capital reallocation and the effects of government spending," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 48(1), pages 145-194, June.
  13. Finn E. Kydland & Edward C. Prescott, 1994. "The computational experiment: an econometric tool," Working Paper 9420, Federal Reserve Bank of Cleveland.
  14. Leamer, Edward E, 1985. "Sensitivity Analyses Would Help," American Economic Review, American Economic Association, vol. 75(3), pages 308-13, June.
  15. Heckman, James J. & Urzua, Sergio, 2009. "Comparing IV with Structural Models: What Simple IV Can and Cannot Identify," IZA Discussion Papers 3980, Institute for the Study of Labor (IZA).
  16. David Card, 1988. "Unexpected Inflation, Real Wages, and Employment Determination in Union Contracts," Working Papers 612, Princeton University, Department of Economics, Industrial Relations Section..
  17. Nunn, Nathan, 2007. "The Long-Term Effects of Africa's Slave Trades," MPRA Paper 4134, University Library of Munich, Germany.
  18. Passell, Peter & Taylor, John B, 1977. "The Deterrent Effect of Capital Punishment: Another View," American Economic Review, American Economic Association, vol. 67(3), pages 445-51, June.
  19. McCrary, Justin, 2008. "Manipulation of the running variable in the regression discontinuity design: A density test," Journal of Econometrics, Elsevier, vol. 142(2), pages 698-714, February.
  20. Donohue, John J & Wolfers, Justin, 2006. "Uses and Abuses of Empirical Evidence in the Death Penalty Debate," CEPR Discussion Papers 5493, C.E.P.R. Discussion Papers.
  21. Orley Ashenfelter & Daniel Hosken, 2008. "The Effect of Mergers on Consumer Prices: Evidence from Five Selected Case Studies," NBER Working Papers 13859, National Bureau of Economic Research, Inc.
  22. Olivier Deschênes & Michael Greenstone, 2007. "Climate Change, Mortality, and Adaptation: Evidence from Annual Fluctuations in Weather in the US," Working Papers 0707, Massachusetts Institute of Technology, Center for Energy and Environmental Policy Research.
  23. Robert J. Barro & Charles J. Redlick, 2011. "Macroeconomic Effects From Government Purchases and Taxes," The Quarterly Journal of Economics, Oxford University Press, vol. 126(1), pages 51-102.
  24. Miguel Urquiola & Eric A. Verhoogen, 2007. "Class size and sorting in market equilibrium: Theory and evidence," Discussion Papers 0607-14, Columbia University, Department of Economics.
  25. White, Halbert, 1980. "Using Least Squares to Approximate Unknown Regression Functions," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 21(1), pages 149-70, February.
  26. Ehrlich, Isaac, 1975. "The Deterrent Effect of Capital Punishment: A Question of Life and Death," American Economic Review, American Economic Association, vol. 65(3), pages 397-417, June.
  27. Dani Rodrik & Romain Wacziarg, 2005. "Do Democratic Transitions Produce Bad Economic Outcomes?," American Economic Review, American Economic Association, vol. 95(2), pages 50-55, May.
  28. Guido W. Imbens, 2009. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," NBER Working Papers 14896, National Bureau of Economic Research, Inc.
  29. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
  30. James Heckman, 1997. "Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 441-462.
  31. Albouy, David, 2006. "The Colonial Origins of Comparative Development: An Investigation of the Settler Mortality Data," Center for International and Development Economics Research, Working Paper Series qt8kt576x8, Center for International and Development Economics Research, Institute for Business and Economic Research, UC Berkeley.
  32. McAleer, Michael & Pagan, Adrian R & Volker, Paul A, 1985. "What Will Take the Con out of Econometrics?," American Economic Review, American Economic Association, vol. 75(3), pages 293-307, June.
  33. Feldstein, Martin & Horioka, Charles, 1980. "Domestic Saving and International Capital Flows," Economic Journal, Royal Economic Society, vol. 90(358), pages 314-29, June.
  34. Orley Ashenfelter, 1990. "Non-Parametric Estimates of the Labor Supply Effects of Negative Income Tax Programs," Working Papers 639, Princeton University, Department of Economics, Industrial Relations Section..
  35. Hausman, Jerry A & Leonard, Gregory K, 2002. "The Competitive Effects of a New Product Introduction: A Case Study," Journal of Industrial Economics, Wiley Blackwell, vol. 50(3), pages 237-63, September.
  36. Angrist, Joshua D, 1990. "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records," American Economic Review, American Economic Association, vol. 80(3), pages 313-36, June.
  37. François R. Velde, 2009. "Chronicle of a Deflation Unforetold," Journal of Political Economy, University of Chicago Press, vol. 117(4), pages 591-634, 08.
  38. Orley C. Ashenfelter & Daniel Hosken & Matthew Weinberg, 2009. "Generating Evidence to Guide Merger Enforcement," NBER Working Papers 14798, National Bureau of Economic Research, Inc.
  39. Kydland, Finn E & Prescott, Edward C, 1977. "Rules Rather Than Discretion: The Inconsistency of Optimal Plans," Journal of Political Economy, University of Chicago Press, vol. 85(3), pages 473-91, June.
  40. Justine S. Hastings, 2004. "Vertical Relationships and Competition in Retail Gasoline Markets: Empirical Evidence from Contract Changes in Southern California," American Economic Review, American Economic Association, vol. 94(1), pages 317-328, March.
  41. Ashenfelter, Orley, 1987. "The case for evaluating training programs with randomized trials," Economics of Education Review, Elsevier, vol. 6(4), pages 333-338, August.
  42. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
  43. Ernst Fehr & Lorenz Goette, 2002. "Do workers work more if wages are high? Evidence from a randomized field experiment," Natural Field Experiments 00240, The Field Experiments Website.
  44. Gary Solon, 1984. "Work Incentive Effects of Taxing Unemployment Benefits," NBER Working Papers 1260, National Bureau of Economic Research, Inc.
  45. Dean Karlan & Jonathan Zinman, 2005. "Observing unobservables: identifying information asymmetries with a consumer-credit field experiment," Proceedings 961, Federal Reserve Bank of Chicago.
  46. Michael Conlin & Ted O'Donoghue & Timothy J. Vogelsang, 2007. "Projection Bias in Catalog Orders," American Economic Review, American Economic Association, vol. 97(4), pages 1217-1249, September.
  47. Dean S. Karlan & Jonathan Zinman, 2005. "Observing Unobservables: Identifying Information Asymmetries with a Consumer Credit Field Experiment," Working Papers 911, Economic Growth Center, Yale University.
  48. Milton Friedman & Anna J. Schwartz, 1963. "A Monetary History of the United States, 1867-1960," NBER Books, National Bureau of Economic Research, Inc, number frie63-1.
  49. Joshua D. Angrist & Victor Lavy, 1999. "Using Maimonides' Rule To Estimate The Effect Of Class Size On Scholastic Achievement," The Quarterly Journal of Economics, MIT Press, vol. 114(2), pages 533-575, May.
  50. Steven G. Rivkin & Eric A. Hanushek & John F. Kain, 2005. "Teachers, Schools, and Academic Achievement," Econometrica, Econometric Society, vol. 73(2), pages 417-458, 03.
  51. Gerald S. Oettinger, 1999. "An Empirical Analysis of the Daily Labor Supply of Stadium Vendors," Journal of Political Economy, University of Chicago Press, vol. 107(2), pages 360-392, April.
  52. Jonathan B. Baker, 2003. "The Case for Antitrust Enforcement," Journal of Economic Perspectives, American Economic Association, vol. 17(4), pages 27-50, Fall.
  53. Robert W. Crandall & Clifford Winston, 2003. "Does Antitrust Policy Improve Consumer Welfare? Assessing the Evidence," Journal of Economic Perspectives, American Economic Association, vol. 17(4), pages 3-26, Fall.
  54. Sims, Christopher A, 1988. "Uncertainty across Models," American Economic Review, American Economic Association, vol. 78(2), pages 163-67, May.
  55. Benjamin J. Keys & Tanmoy Mukherjee & Amit Seru & Vikrant Vig, 2010. "Did Securitization Lead to Lax Screening? Evidence from Subprime Loans," The Quarterly Journal of Economics, MIT Press, vol. 125(1), pages 307-362, February.
  56. Summers, Anita A & Wolfe, Barbara L, 1977. "Do Schools Make a Difference?," American Economic Review, American Economic Association, vol. 67(4), pages 639-52, September.
  57. Hoyt Bleakley, 2007. "Disease and Development: Evidence from Hookworm Eradication in the American South," The Quarterly Journal of Economics, MIT Press, vol. 122(1), pages 73-117, 02.
  58. Stefano DellaVigna & M. Daniele Paserman, 2004. "Job Search and Impatience," NBER Working Papers 10837, National Bureau of Economic Research, Inc.
  59. Aviv Nevo, 2000. "Mergers with Differentiated Products: The Case of the Ready-to-Eat Cereal Industry," RAND Journal of Economics, The RAND Corporation, vol. 31(3), pages 395-421, Autumn.
  60. repec:cpi:cpijrn:5.1.2009:i=5240 is not listed on IDEAS
  61. Ehrlich, Isaac & Brower, George D, 1987. "On the Issue of Causality in the Economic Model of Crime and Law Enforcement: Some Theoretical Considerations and Experimental Evidence," American Economic Review, American Economic Association, vol. 77(2), pages 99-106, May.
  62. Angrist, Joshua D, 1990. "Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records: Errata," American Economic Review, American Economic Association, vol. 80(5), pages 1284-86, December.
  63. Caroline M. Hoxby, 2000. "The Effects Of Class Size On Student Achievement: New Evidence From Population Variation," The Quarterly Journal of Economics, MIT Press, vol. 115(4), pages 1239-1285, November.
  64. Ehrlich, Isaac, 1977. "Capital Punishment and Deterrence: Some Further Thoughts and Additional Evidence," Journal of Political Economy, University of Chicago Press, vol. 85(4), pages 741-88, August.
  65. Stefano DellaVigna & Ulrike Malmendier, 2006. "Paying Not to Go to the Gym," American Economic Review, American Economic Association, vol. 96(3), pages 694-719, June.
  66. Brian A. Jacob, 2004. "Public Housing, Housing Vouchers, and Student Achievement: Evidence from Public Housing Demolitions in Chicago," American Economic Review, American Economic Association, vol. 94(1), pages 233-258, March.
  67. Hanushek, Eric A, 1986. "The Economics of Schooling: Production and Efficiency in Public Schools," Journal of Economic Literature, American Economic Association, vol. 24(3), pages 1141-77, September.
  68. Joshua D. Angrist & Guido M. Kuersteiner, 2004. "Semiparametric Causality Tests Using the Policy Propensity Score," NBER Working Papers 10975, National Bureau of Economic Research, Inc.
  69. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  70. Isaac Ehrlich, 1996. "Crime, Punishment, and the Market for Offenses," Journal of Economic Perspectives, American Economic Association, vol. 10(1), pages 43-67, Winter.
  71. Greenberg, David & Halsey, Harlan, 1983. "Systematic Misreporting and Effects of Income Maintenance Experiments on Work Effort: Evidence from the Seattle-Denver Experiment," Journal of Labor Economics, University of Chicago Press, vol. 1(4), pages 380-407, October.
  72. Ehrlich, Isaac & Liu, Zhiqiang, 1999. "Sensitivity Analyses of the Deterrence Hypothesis: Let's Keep the Econ in Econometrics," Journal of Law and Economics, University of Chicago Press, vol. 42(1), pages 455-87, April.
  73. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-38, May.
  74. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
  75. Manning, Willard G, et al, 1987. "Health Insurance and the Demand for Medical Care: Evidence from a Randomized Experiment," American Economic Review, American Economic Association, vol. 77(3), pages 251-77, June.
  76. Borenstein, Severin, 1990. "Airline Mergers, Airport Dominance, and Market Power," American Economic Review, American Economic Association, vol. 80(2), pages 400-404, May.
  77. Gary Richardson & William Troost, 2009. "Monetary Intervention Mitigated Banking Panics during the Great Depression: Quasi-Experimental Evidence from a Federal Reserve District Border, 1929-1933," Journal of Political Economy, University of Chicago Press, vol. 117(6), pages 1031-1073, December.
  78. Cecilia Elena Rouse, 1998. "Private School Vouchers And Student Achievement: An Evaluation Of The Milwaukee Parental Choice Program," The Quarterly Journal of Economics, MIT Press, vol. 113(2), pages 553-602, May.
  79. Paul Gertler, 2004. "Do Conditional Cash Transfers Improve Child Health? Evidence from PROGRESA's Control Randomized Experiment," American Economic Review, American Economic Association, vol. 94(2), pages 336-341, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:24:y:2010:i:2:p:3-30. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jane Voros)

or (Michael P. Albert)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.