IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this article or follow this journal

Robustness checks and robustness tests in applied economics

  • Lu, Xun
  • White, Halbert

A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. As we show, there are numerous pitfalls, as commonly implemented robustness checks give neither necessary nor sufficient evidence for structural validity. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. We discuss how critical and non-critical core variables can be properly specified and how non-core variables for the comparison regression can be chosen to ensure that robustness checks are indeed structurally informative. We provide a straightforward new Hausman (1978) type test of robustness for the critical core coefficients, additional diagnostics that can help explain why robustness test rejection occurs, and a new estimator, the Feasible Optimally combined GLS (FOGLeSs) estimator, that makes relatively efficient use of the robustness check regressions. A new procedure for Matlab, testrob, embodies these methods.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.sciencedirect.com/science/article/pii/S0304407613001668
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Article provided by Elsevier in its journal Journal of Econometrics.

Volume (Year): 178 (2014)
Issue (Month): P1 ()
Pages: 194-206

as
in new window

Handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206
Contact details of provider: Web page: http://www.elsevier.com/locate/jeconom

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. William Adams & Liran Einav & Jonathan Levin, 2009. "Liquidity Constraints and Imperfect Information in Subprime Lending," American Economic Review, American Economic Association, vol. 99(1), pages 49-84, March.
  2. Halbert White & Xun Lu, 2010. "Granger Causality and Dynamic Structural Systems," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 8(2), pages 193-243, spring.
  3. Gallant, A. Ronald, 1982. "Unbiased determination of production technologies," Journal of Econometrics, Elsevier, vol. 20(2), pages 285-323, November.
  4. Looney, Adam & Kroft, Kory & Chetty, Raj, 2009. "Salience and Taxation: Theory and Evidence," Scholarly Articles 9748525, Harvard University Department of Economics.
  5. Mas, Alexandre & Moretti, Enrico, 2006. "Peers at Work," CEPR Discussion Papers 5870, C.E.P.R. Discussion Papers.
  6. Felix Oberholzer-Gee & Joel Waldfogel, 2006. "Media Markets and Localism: Does Local News en Español Boost Hispanic Voter Turnout?," NBER Working Papers 12317, National Bureau of Economic Research, Inc.
  7. Francisco Pérez-González, 2006. "Inherited Control and Firm Performance," American Economic Review, American Economic Association, vol. 96(5), pages 1559-1588, December.
  8. Spilimbergo, Antonio, 2006. "Democracy and Foreign Education," CEPR Discussion Papers 5934, C.E.P.R. Discussion Papers.
  9. Hongbin Cai & Yuyu Chen & Hanming Fang, 2009. "Observational Learning: Evidence from a Randomized Natural Field Experiment," American Economic Review, American Economic Association, vol. 99(3), pages 864-82, June.
  10. Su, Liangjun & White, Halbert, 2010. "Testing Structural Change In Partially Linear Models," Econometric Theory, Cambridge University Press, vol. 26(06), pages 1761-1806, December.
  11. Hamilton, James D, 2001. "A Parametric Approach to Flexible Nonlinear Inference," Econometrica, Econometric Society, vol. 69(3), pages 537-73, May.
  12. Guido W. Imbens, 2003. "Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review," NBER Technical Working Papers 0294, National Bureau of Economic Research, Inc.
  13. Yan Chen & Sherry Xin Li, 2009. "Group Identity and Social Preferences," American Economic Review, American Economic Association, vol. 99(1), pages 431-57, March.
  14. Urquiola, Miguel & Verhoogen, Eric A, 2007. "Class Size and Sorting in Market Equilibrium: Theory and Evidence," CEPR Discussion Papers 6425, C.E.P.R. Discussion Papers.
  15. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
  16. Laura Alfaro & Andrew Charlton, 2009. "Intra-industry Foreign Direct Investment," American Economic Review, American Economic Association, vol. 99(5), pages 2096-2119, December.
  17. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
  18. Jean Boivin & Marc P. Giannoni & Ilian Mihov, 2009. "Sticky Prices and Monetary Policy: Evidence from Disaggregated US Data," American Economic Review, American Economic Association, vol. 99(1), pages 350-84, March.
  19. Manuela Angelucci & Giacomo De Giorgi, 2009. "Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?," American Economic Review, American Economic Association, vol. 99(1), pages 486-508, March.
  20. Shaun P. Hargreaves Heap & Daniel John Zizzo, 2009. "The Value of Groups," American Economic Review, American Economic Association, vol. 99(1), pages 295-323, March.
  21. Igal Hendel & Aviv Nevo & François Ortalo-Magné, 2009. "The Relative Performance of Real Estate Marketing Platforms: MLS versus FSBOMadison.com," American Economic Review, American Economic Association, vol. 99(5), pages 1878-98, December.
  22. Silke Januszewski Forbes & Mara Lederman, 2009. "Adaptation and Vertical Integration in the Airline Industry," American Economic Review, American Economic Association, vol. 99(5), pages 1831-49, December.
  23. Clemens Sialm, 2009. "Tax Changes and Asset Pricing," American Economic Review, American Economic Association, vol. 99(4), pages 1356-83, September.
  24. Griliches, Zvi, 1977. "Estimating the Returns to Schooling: Some Econometric Problems," Econometrica, Econometric Society, vol. 45(1), pages 1-22, January.
  25. Andrews, Donald W K, 1994. "Asymptotics for Semiparametric Econometric Models via Stochastic Equicontinuity," Econometrica, Econometric Society, vol. 62(1), pages 43-72, January.
  26. Karim Chalak & Halbert White, 2008. "Causality, Conditional Independence, and Graphical Separation in Settable Systems," Boston College Working Papers in Economics 689, Boston College Department of Economics, revised 04 Jul 2010.
  27. White, Halbert, 2006. "Approximate Nonlinear Forecasting Methods," Handbook of Economic Forecasting, Elsevier.
  28. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
  29. J. A. Hausman, 1976. "Specification Tests in Econometrics," Working papers 185, Massachusetts Institute of Technology (MIT), Department of Economics.
  30. White, Halbert, 2006. "Time-series estimation of the effects of natural experiments," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 527-566.
  31. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
  32. Victor Lavy, 2004. "Performance Pay and Teachers' Effort, Productivity and Grading Ethics," NBER Working Papers 10622, National Bureau of Economic Research, Inc.
  33. John G. Matsusaka, 2009. "Direct Democracy and Public Employees," American Economic Review, American Economic Association, vol. 99(5), pages 2227-46, December.
  34. Halbert White & Karim Chalak, 2010. "Testing a Conditional Form of Exogeneity," Boston College Working Papers in Economics 733, Boston College Department of Economics.
  35. Karim Chalak & Halbert White, 2011. "Viewpoint: An extended class of instrumental variables for the estimation of causal effects," Canadian Journal of Economics, Canadian Economics Association, vol. 44(1), pages 1-51, February.
  36. Jinyong Hahn, 2004. "Functional Restriction and Efficiency in Causal Inference," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 73-76, February.
  37. Nathan H. Miller, 2009. "Strategic Leniency and Cartel Enforcement," American Economic Review, American Economic Association, vol. 99(3), pages 750-68, June.
  38. Michael D. Makowsky & Thomas Stratmann, 2009. "Political Economy at Any Speed: What Determines Traffic Citations?," American Economic Review, American Economic Association, vol. 99(1), pages 509-27, March.
  39. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
  40. Clare Leaver, 2009. "Bureaucratic Minimal Squawk Behavior: Theory and Evidence from Regulatory Agencies," American Economic Review, American Economic Association, vol. 99(3), pages 572-607, June.
  41. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-77, September.
  42. Halbert White & Xun Lu, 2011. "Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1453-1459, November.
  43. Robinson, P M, 1987. "Asymptotically Efficient Estimation in the Presence of Heteroskedasticity of Unknown Form," Econometrica, Econometric Society, vol. 55(4), pages 875-91, July.
  44. Lee, Tae-Hwy & White, Halbert & Granger, Clive W. J., 1993. "Testing for neglected nonlinearity in time series models : A comparison of neural network methods and alternative tests," Journal of Econometrics, Elsevier, vol. 56(3), pages 269-290, April.
  45. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-38, May.
  46. Carlos Dobkin & Nancy Nicosia, 2009. "The War on Drugs: Methamphetamine, Public Health, and Crime," American Economic Review, American Economic Association, vol. 99(1), pages 324-49, March.
  47. Dahl, Christian M. & Gonzalez-Rivera, Gloria, 2003. "Testing for neglected nonlinearity in regression models based on the theory of random fields," Journal of Econometrics, Elsevier, vol. 114(1), pages 141-164, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.