IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this article or follow this journal

Robustness checks and robustness tests in applied economics

  • Lu, Xun
  • White, Halbert

A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. As we show, there are numerous pitfalls, as commonly implemented robustness checks give neither necessary nor sufficient evidence for structural validity. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. We discuss how critical and non-critical core variables can be properly specified and how non-core variables for the comparison regression can be chosen to ensure that robustness checks are indeed structurally informative. We provide a straightforward new Hausman (1978) type test of robustness for the critical core coefficients, additional diagnostics that can help explain why robustness test rejection occurs, and a new estimator, the Feasible Optimally combined GLS (FOGLeSs) estimator, that makes relatively efficient use of the robustness check regressions. A new procedure for Matlab, testrob, embodies these methods.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.sciencedirect.com/science/article/pii/S0304407613001668
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Article provided by Elsevier in its journal Journal of Econometrics.

Volume (Year): 178 (2014)
Issue (Month): P1 ()
Pages: 194-206

as
in new window

Handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206
Contact details of provider: Web page: http://www.elsevier.com/locate/jeconom

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
  2. Robinson, P M, 1987. "Asymptotically Efficient Estimation in the Presence of Heteroskedasticity of Unknown Form," Econometrica, Econometric Society, vol. 55(4), pages 875-91, July.
  3. Clemens Sialm, 2009. "Tax Changes and Asset Pricing," American Economic Review, American Economic Association, vol. 99(4), pages 1356-83, September.
  4. Mas, Alexandre & Moretti, Enrico, 2006. "Peers at Work," CEPR Discussion Papers 5870, C.E.P.R. Discussion Papers.
  5. Andrews, Donald W K, 1994. "Asymptotics for Semiparametric Econometric Models via Stochastic Equicontinuity," Econometrica, Econometric Society, vol. 62(1), pages 43-72, January.
  6. Jean Boivin & Marc Giannoni & Ilian Mihov, 2007. "Sticky Prices and Monetary Policy: Evidence from Disaggregated U.S. Data," NBER Working Papers 12824, National Bureau of Economic Research, Inc.
  7. Antonio Spilimbergo, 2009. "Democracy and Foreign Education," American Economic Review, American Economic Association, vol. 99(1), pages 528-43, March.
  8. Urquiola, Miguel & Verhoogen, Eric A, 2007. "Class Size and Sorting in Market Equilibrium: Theory and Evidence," CEPR Discussion Papers 6425, C.E.P.R. Discussion Papers.
  9. Hausman, Jerry A, 1978. "Specification Tests in Econometrics," Econometrica, Econometric Society, vol. 46(6), pages 1251-71, November.
  10. Lee, Tae-Hwy & White, Halbert & Granger, Clive W. J., 1993. "Testing for neglected nonlinearity in time series models : A comparison of neural network methods and alternative tests," Journal of Econometrics, Elsevier, vol. 56(3), pages 269-290, April.
  11. Michael D. Makowsky & Thomas Stratmann, 2009. "Political Economy at Any Speed: What Determines Traffic Citations?," American Economic Review, American Economic Association, vol. 99(1), pages 509-27, March.
  12. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
  13. William Adams & Liran Einav & Jonathan Levin, 2007. "Liquidity Constraints and Imperfect Information in Subprime Lending," NBER Working Papers 13067, National Bureau of Economic Research, Inc.
  14. Laura Alfaro & Andrew Charlton, 2009. "Intra-industry Foreign Direct Investment," American Economic Review, American Economic Association, vol. 99(5), pages 2096-2119, December.
  15. James J. Heckman, 1989. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training," NBER Working Papers 2861, National Bureau of Economic Research, Inc.
  16. Shaun P. Hargreaves Heap & Daniel John Zizzo, 2009. "The Value of Groups," American Economic Review, American Economic Association, vol. 99(1), pages 295-323, March.
  17. Guido W. Imbens, 2003. "Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review," NBER Technical Working Papers 0294, National Bureau of Economic Research, Inc.
  18. Halbert White & Xun Lu, 2010. "Granger Causality and Dynamic Structural Systems," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 8(2), pages 193-243, spring.
  19. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
  20. Heckman, James J. & Navarro, Salvador, 2003. "Using Matching, Instrumental Variables and Control Functions to Estimate Economic Choice Models," IZA Discussion Papers 768, Institute for the Study of Labor (IZA).
  21. Hongbin Cai & Yuyu Chen & Hanming Fang, 2007. "Observational Learning: Evidence from a Randomized Natural Field Experiment," NBER Working Papers 13516, National Bureau of Economic Research, Inc.
  22. Halbert White & Karim Chalak, 2010. "Testing a Conditional Form of Exogeneity," Boston College Working Papers in Economics 733, Boston College Department of Economics.
  23. Su, Liangjun & White, Halbert, 2010. "Testing Structural Change In Partially Linear Models," Econometric Theory, Cambridge University Press, vol. 26(06), pages 1761-1806, December.
  24. Carlos Dobkin & Nancy Nicosia, 2009. "The War on Drugs: Methamphetamine, Public Health, and Crime," American Economic Review, American Economic Association, vol. 99(1), pages 324-49, March.
  25. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-38, May.
  26. Raj Chetty & Adam Looney & Kory Kroft, 2009. "Salience and Taxation: Theory and Evidence," American Economic Review, American Economic Association, vol. 99(4), pages 1145-77, September.
  27. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
  28. Hamilton, James D., 1999. "A Parametric Approach to Flexible Nonlinear Inference," University of California at San Diego, Economics Working Paper Series qt68s8157x, Department of Economics, UC San Diego.
  29. Francisco Pérez-González, 2006. "Inherited Control and Firm Performance," American Economic Review, American Economic Association, vol. 96(5), pages 1559-1588, December.
  30. Halbert White & Xun Lu, 2011. "Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1453-1459, November.
  31. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-77, September.
  32. White, Halbert, 2006. "Approximate Nonlinear Forecasting Methods," Handbook of Economic Forecasting, Elsevier.
  33. Silke Januszewski Forbes & Mara Lederman, 2009. "Adaptation and Vertical Integration in the Airline Industry," American Economic Review, American Economic Association, vol. 99(5), pages 1831-49, December.
  34. Yan Chen & Sherry Xin Li, 2009. "Group Identity and Social Preferences," American Economic Review, American Economic Association, vol. 99(1), pages 431-57, March.
  35. Igal Hendel & Aviv Nevo & François Ortalo-Magné, 2009. "The Relative Performance of Real Estate Marketing Platforms: MLS versus FSBOMadison.com," American Economic Review, American Economic Association, vol. 99(5), pages 1878-98, December.
  36. Jinyong Hahn, 2004. "Functional Restriction and Efficiency in Causal Inference," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 73-76, February.
  37. Karim Chalak & Halbert White, 2011. "Viewpoint: An extended class of instrumental variables for the estimation of causal effects," Canadian Journal of Economics, Canadian Economics Association, vol. 44(1), pages 1-51, February.
  38. Gallant, A. Ronald, 1982. "Unbiased determination of production technologies," Journal of Econometrics, Elsevier, vol. 20(2), pages 285-323, November.
  39. Clare Leaver, 2009. "Bureaucratic Minimal Squawk Behavior: Theory and Evidence from Regulatory Agencies," American Economic Review, American Economic Association, vol. 99(3), pages 572-607, June.
  40. Griliches, Zvi, 1977. "Estimating the Returns to Schooling: Some Econometric Problems," Econometrica, Econometric Society, vol. 45(1), pages 1-22, January.
  41. Felix Oberholzer-Gee & Joel Waldfogel, 2009. "Media Markets and Localism: Does Local News en Español Boost Hispanic Voter Turnout?," American Economic Review, American Economic Association, vol. 99(5), pages 2120-28, December.
  42. Manuela Angelucci & Giacomo De Giorgi, 2009. "Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?," American Economic Review, American Economic Association, vol. 99(1), pages 486-508, March.
  43. Karim Chalak & Halbert White, 2008. "Causality, Conditional Independence, and Graphical Separation in Settable Systems," Boston College Working Papers in Economics 689, Boston College Department of Economics, revised 04 Jul 2010.
  44. John G. Matsusaka, 2009. "Direct Democracy and Public Employees," American Economic Review, American Economic Association, vol. 99(5), pages 2227-46, December.
  45. Nathan H. Miller, 2009. "Strategic Leniency and Cartel Enforcement," American Economic Review, American Economic Association, vol. 99(3), pages 750-68, June.
  46. White, Halbert, 2006. "Time-series estimation of the effects of natural experiments," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 527-566.
  47. Dahl, Christian M. & Gonzalez-Rivera, Gloria, 2003. "Testing for neglected nonlinearity in regression models based on the theory of random fields," Journal of Econometrics, Elsevier, vol. 114(1), pages 141-164, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.