IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this article or follow this journal

Robustness checks and robustness tests in applied economics

  • Lu, Xun
  • White, Halbert

A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. As we show, there are numerous pitfalls, as commonly implemented robustness checks give neither necessary nor sufficient evidence for structural validity. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. We discuss how critical and non-critical core variables can be properly specified and how non-core variables for the comparison regression can be chosen to ensure that robustness checks are indeed structurally informative. We provide a straightforward new Hausman (1978) type test of robustness for the critical core coefficients, additional diagnostics that can help explain why robustness test rejection occurs, and a new estimator, the Feasible Optimally combined GLS (FOGLeSs) estimator, that makes relatively efficient use of the robustness check regressions. A new procedure for Matlab, testrob, embodies these methods.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.sciencedirect.com/science/article/pii/S0304407613001668
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Article provided by Elsevier in its journal Journal of Econometrics.

Volume (Year): 178 (2014)
Issue (Month): P1 ()
Pages: 194-206

as
in new window

Handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206
Contact details of provider: Web page: http://www.elsevier.com/locate/jeconom

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Dahl, Christian M. & Gonzalez-Rivera, Gloria, 2003. "Testing for neglected nonlinearity in regression models based on the theory of random fields," Journal of Econometrics, Elsevier, vol. 114(1), pages 141-164, May.
  2. Griliches, Zvi, 1977. "Estimating the Returns to Schooling: Some Econometric Problems," Econometrica, Econometric Society, vol. 45(1), pages 1-22, January.
  3. Clare Leaver, 2009. "Bureaucratic Minimal Squawk Behavior: Theory and Evidence from Regulatory Agencies," American Economic Review, American Economic Association, vol. 99(3), pages 572-607, June.
  4. White, Halbert & Chalak, Karim, 2010. "Testing a conditional form of exogeneity," Economics Letters, Elsevier, vol. 109(2), pages 88-90, November.
  5. Laura Alfaro & Andrew Charlton, 2009. "Intra-industry Foreign Direct Investment," American Economic Review, American Economic Association, vol. 99(5), pages 2096-2119, December.
  6. Jean Boivin & Marc Giannoni & Ilian Mihov, 2007. "Sticky Prices and Monetary Policy: Evidence from Disaggregated U.S. Data," NBER Working Papers 12824, National Bureau of Economic Research, Inc.
  7. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
  8. Halbert White & Xun Lu, 2011. "Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1453-1459, November.
  9. John G. Matsusaka, 2009. "Direct Democracy and Public Employees," American Economic Review, American Economic Association, vol. 99(5), pages 2227-46, December.
  10. Francisco Pérez-González, 2006. "Inherited Control and Firm Performance," American Economic Review, American Economic Association, vol. 96(5), pages 1559-1588, December.
  11. Miguel Urquiola & Eric Verhoogen, 2009. "Class-Size Caps, Sorting, and the Regression-Discontinuity Design," American Economic Review, American Economic Association, vol. 99(1), pages 179-215, March.
  12. Clemens Sialm, 2009. "Tax Changes and Asset Pricing," American Economic Review, American Economic Association, vol. 99(4), pages 1356-83, September.
  13. Alexandre Mas & Enrico Moretti, 2006. "Peers at Work," NBER Working Papers 12508, National Bureau of Economic Research, Inc.
  14. Antonio Spilimbergo, 2009. "Democracy and Foreign Education," American Economic Review, American Economic Association, vol. 99(1), pages 528-43, March.
  15. Francois Ortalo-Magne & Aviv Nevo & Igal Hendel, 2007. "The Relative Performance of Real Estate Marketing Platforms: MLS versus FSBOMadison.com," 2007 Meeting Papers 89, Society for Economic Dynamics.
  16. Raj Chetty & Adam Looney & Kory Kroft, 2007. "Salience and Taxation: Theory and Evidence," NBER Working Papers 13330, National Bureau of Economic Research, Inc.
  17. William Adams & Liran Einav & Jonathan Levin, 2007. "Liquidity Constraints and Imperfect Information in Subprime Lending," NBER Working Papers 13067, National Bureau of Economic Research, Inc.
  18. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
  19. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
  20. Gallant, A. Ronald, 1982. "Unbiased determination of production technologies," Journal of Econometrics, Elsevier, vol. 20(2), pages 285-323, November.
  21. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-77, September.
  22. Karim Chalak & Halbert White, 2008. "Causality, Conditional Independence, and Graphical Separation in Settable Systems," Boston College Working Papers in Economics 689, Boston College Department of Economics, revised 04 Jul 2010.
  23. Hongbin Cai & Yuyu Chen & Hanming Fang, 2007. "Observational Learning: Evidence from a Randomized Natural Field Experiment," NBER Working Papers 13516, National Bureau of Economic Research, Inc.
  24. Su, Liangjun & White, Halbert, 2010. "Testing Structural Change In Partially Linear Models," Econometric Theory, Cambridge University Press, vol. 26(06), pages 1761-1806, December.
  25. Lee, Tae-Hwy & White, Halbert & Granger, Clive W. J., 1993. "Testing for neglected nonlinearity in time series models : A comparison of neural network methods and alternative tests," Journal of Econometrics, Elsevier, vol. 56(3), pages 269-290, April.
  26. J. A. Hausman, 1976. "Specification Tests in Econometrics," Working papers 185, Massachusetts Institute of Technology (MIT), Department of Economics.
  27. Halbert White & Xun Lu, 2010. "Granger Causality and Dynamic Structural Systems," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 8(2), pages 193-243, spring.
  28. Yan Chen & Sherry Xin Li, 2009. "Group Identity and Social Preferences," American Economic Review, American Economic Association, vol. 99(1), pages 431-57, March.
  29. Hamilton, James D., 1999. "A Parametric Approach to Flexible Nonlinear Inference," University of California at San Diego, Economics Working Paper Series qt68s8157x, Department of Economics, UC San Diego.
  30. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
  31. Shaun P. Hargreaves Heap & Daniel John Zizzo, 2009. "The Value of Groups," American Economic Review, American Economic Association, vol. 99(1), pages 295-323, March.
  32. Silke Januszewski Forbes & Mara Lederman, 2009. "Adaptation and Vertical Integration in the Airline Industry," American Economic Review, American Economic Association, vol. 99(5), pages 1831-49, December.
  33. White, Halbert, 2006. "Approximate Nonlinear Forecasting Methods," Handbook of Economic Forecasting, Elsevier.
  34. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
  35. Felix Oberholzer-Gee & Joel Waldfogel, 2009. "Media Markets and Localism: Does Local News en Español Boost Hispanic Voter Turnout?," American Economic Review, American Economic Association, vol. 99(5), pages 2120-28, December.
  36. Nathan H. Miller, 2009. "Strategic Leniency and Cartel Enforcement," American Economic Review, American Economic Association, vol. 99(3), pages 750-68, June.
  37. Michael D. Makowsky & Thomas Stratmann, 2009. "Political Economy at Any Speed: What Determines Traffic Citations?," American Economic Review, American Economic Association, vol. 99(1), pages 509-27, March.
  38. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
  39. Andrews, Donald W K, 1994. "Asymptotics for Semiparametric Econometric Models via Stochastic Equicontinuity," Econometrica, Econometric Society, vol. 62(1), pages 43-72, January.
  40. Manuela Angelucci & Giacomo De Giorgi, 2009. "Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?," American Economic Review, American Economic Association, vol. 99(1), pages 486-508, March.
  41. Jinyong Hahn, 2004. "Functional Restriction and Efficiency in Causal Inference," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 73-76, February.
  42. Carlos Dobkin & Nancy Nicosia, 2009. "The War on Drugs: Methamphetamine, Public Health, and Crime," American Economic Review, American Economic Association, vol. 99(1), pages 324-49, March.
  43. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-38, May.
  44. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
  45. Robinson, P M, 1987. "Asymptotically Efficient Estimation in the Presence of Heteroskedasticity of Unknown Form," Econometrica, Econometric Society, vol. 55(4), pages 875-91, July.
  46. Karim Chalak & Halbert White, 2011. "Viewpoint: An extended class of instrumental variables for the estimation of causal effects," Canadian Journal of Economics, Canadian Economics Association, vol. 44(1), pages 1-51, February.
  47. White, Halbert, 2006. "Time-series estimation of the effects of natural experiments," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 527-566.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.