Advanced Search
MyIDEAS: Login

Robustness checks and robustness tests in applied economics

Contents:

Author Info

  • Lu, Xun
  • White, Halbert

Abstract

A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. As we show, there are numerous pitfalls, as commonly implemented robustness checks give neither necessary nor sufficient evidence for structural validity. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. We discuss how critical and non-critical core variables can be properly specified and how non-core variables for the comparison regression can be chosen to ensure that robustness checks are indeed structurally informative. We provide a straightforward new Hausman (1978) type test of robustness for the critical core coefficients, additional diagnostics that can help explain why robustness test rejection occurs, and a new estimator, the Feasible Optimally combined GLS (FOGLeSs) estimator, that makes relatively efficient use of the robustness check regressions. A new procedure for Matlab, testrob, embodies these methods.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://www.sciencedirect.com/science/article/pii/S0304407613001668
Download Restriction: Full text for ScienceDirect subscribers only

As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

Bibliographic Info

Article provided by Elsevier in its journal Journal of Econometrics.

Volume (Year): 178 (2014)
Issue (Month): P1 ()
Pages: 194-206

as in new window
Handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206

Contact details of provider:
Web page: http://www.elsevier.com/locate/jeconom

Related research

Keywords: Robustness; Causal effect; Conditional exogeneity; Specification test; Combined estimator;

Find related papers by JEL classification:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Laura Alfaro & Andrew Charlton, 2007. "Intra-Industry Foreign Direct Investment," NBER Working Papers 13447, National Bureau of Economic Research, Inc.
  2. Miguel Urquiola & Eric Verhoogen, 2009. "Class-Size Caps, Sorting, and the Regression-Discontinuity Design," American Economic Review, American Economic Association, vol. 99(1), pages 179-215, March.
  3. Alexandre Mas & Enrico Moretti, 2006. "Peers at Work," NBER Working Papers 12508, National Bureau of Economic Research, Inc.
  4. Raj Chetty & Adam Looney & Kory Kroft, 2009. "Salience and taxation: theory and evidence," Finance and Economics Discussion Series 2009-11, Board of Governors of the Federal Reserve System (U.S.).
  5. Felix Oberholzer-Gee & Joel Waldfogel, 2006. "Media Markets and Localism: Does Local News en Español Boost Hispanic Voter Turnout?," NBER Working Papers 12317, National Bureau of Economic Research, Inc.
  6. Spilimbergo, Antonio, 2006. "Democracy and Foreign Education," CEPR Discussion Papers 5934, C.E.P.R. Discussion Papers.
  7. Jean Boivin & Marc Giannoni & Ilian Mihov, 2007. "Sticky Prices and Monetary Policy: Evidence from Disaggregated U.S. Data," NBER Working Papers 12824, National Bureau of Economic Research, Inc.
  8. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
  9. Hausman, Jerry A, 1978. "Specification Tests in Econometrics," Econometrica, Econometric Society, vol. 46(6), pages 1251-71, November.
  10. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
  11. Heckman, James J. & Navarro, Salvador, 2003. "Using Matching, Instrumental Variables and Control Functions to Estimate Economic Choice Models," IZA Discussion Papers 768, Institute for the Study of Labor (IZA).
  12. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
  13. Silke Januszewski Forbes & Mara Lederman, 2009. "Adaptation and Vertical Integration in the Airline Industry," American Economic Review, American Economic Association, vol. 99(5), pages 1831-49, December.
  14. Dahl, Christian M. & Gonzalez-Rivera, Gloria, 2003. "Testing for neglected nonlinearity in regression models based on the theory of random fields," Journal of Econometrics, Elsevier, vol. 114(1), pages 141-164, May.
  15. White, Halbert, 2006. "Approximate Nonlinear Forecasting Methods," Handbook of Economic Forecasting, Elsevier.
  16. Su, Liangjun & White, Halbert, 2010. "Testing Structural Change In Partially Linear Models," Econometric Theory, Cambridge University Press, vol. 26(06), pages 1761-1806, December.
  17. James J. Heckman, 1989. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training," NBER Working Papers 2861, National Bureau of Economic Research, Inc.
  18. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
  19. Halbert White & Xun Lu, 2011. "Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1453-1459, November.
  20. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
  21. Hongbin Cai & Yuyu Chen & Hanming Fang, 2009. "Observational Learning: Evidence from a Randomized Natural Field Experiment," American Economic Review, American Economic Association, vol. 99(3), pages 864-82, June.
  22. Gallant, A. Ronald, 1982. "Unbiased determination of production technologies," Journal of Econometrics, Elsevier, vol. 20(2), pages 285-323, November.
  23. Hamilton, James D, 2001. "A Parametric Approach to Flexible Nonlinear Inference," Econometrica, Econometric Society, vol. 69(3), pages 537-73, May.
  24. White, Halbert, 2006. "Time-series estimation of the effects of natural experiments," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 527-566.
  25. Manuela Angelucci & Giacomo De Giorgi, 2009. "Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?," American Economic Review, American Economic Association, vol. 99(1), pages 486-508, March.
  26. Clemens Sialm, 2009. "Tax Changes and Asset Pricing," American Economic Review, American Economic Association, vol. 99(4), pages 1356-83, September.
  27. John G. Matsusaka, 2009. "Direct Democracy and Public Employees," American Economic Review, American Economic Association, vol. 99(5), pages 2227-46, December.
  28. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-38, May.
  29. Nathan H. Miller, 2009. "Strategic Leniency and Cartel Enforcement," American Economic Review, American Economic Association, vol. 99(3), pages 750-68, June.
  30. Francois Ortalo-Magne & Aviv Nevo & Igal Hendel, 2007. "The Relative Performance of Real Estate Marketing Platforms: MLS versus FSBOMadison.com," 2007 Meeting Papers 89, Society for Economic Dynamics.
  31. White, Halbert & Chalak, Karim, 2010. "Testing a conditional form of exogeneity," Economics Letters, Elsevier, vol. 109(2), pages 88-90, November.
  32. Andrews, Donald W K, 1994. "Asymptotics for Semiparametric Econometric Models via Stochastic Equicontinuity," Econometrica, Econometric Society, vol. 62(1), pages 43-72, January.
  33. Halbert White & Xun Lu, 2010. "Granger Causality and Dynamic Structural Systems," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 8(2), pages 193-243, spring.
  34. Yan Chen & Sherry Xin Li, 2009. "Group Identity and Social Preferences," American Economic Review, American Economic Association, vol. 99(1), pages 431-57, March.
  35. William Adams & Liran Einav & Jonathan Levin, 2007. "Liquidity Constraints and Imperfect Information in Subprime Lending," NBER Working Papers 13067, National Bureau of Economic Research, Inc.
  36. Griliches, Zvi, 1977. "Estimating the Returns to Schooling: Some Econometric Problems," Econometrica, Econometric Society, vol. 45(1), pages 1-22, January.
  37. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-77, September.
  38. Karim Chalak & Halbert White, 2011. "Viewpoint: An extended class of instrumental variables for the estimation of causal effects," Canadian Journal of Economics, Canadian Economics Association, vol. 44(1), pages 1-51, February.
  39. Jinyong Hahn, 2004. "Functional Restriction and Efficiency in Causal Inference," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 73-76, February.
  40. Lee, Tae-Hwy & White, Halbert & Granger, Clive W. J., 1993. "Testing for neglected nonlinearity in time series models : A comparison of neural network methods and alternative tests," Journal of Econometrics, Elsevier, vol. 56(3), pages 269-290, April.
  41. Michael D. Makowsky & Thomas Stratmann, 2009. "Political Economy at Any Speed: What Determines Traffic Citations?," American Economic Review, American Economic Association, vol. 99(1), pages 509-27, March.
  42. Clare Leaver, 2009. "Bureaucratic Minimal Squawk Behavior: Theory and Evidence from Regulatory Agencies," American Economic Review, American Economic Association, vol. 99(3), pages 572-607, June.
  43. Carlos Dobkin & Nancy Nicosia, 2009. "The War on Drugs: Methamphetamine, Public Health, and Crime," American Economic Review, American Economic Association, vol. 99(1), pages 324-49, March.
  44. Shaun P. Hargreaves Heap & Daniel John Zizzo, 2009. "The Value of Groups," American Economic Review, American Economic Association, vol. 99(1), pages 295-323, March.
  45. Robinson, P M, 1987. "Asymptotically Efficient Estimation in the Presence of Heteroskedasticity of Unknown Form," Econometrica, Econometric Society, vol. 55(4), pages 875-91, July.
Full references (including those not matched with items on IDEAS)

Citations

Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Tom Chang & Joshua S. Graff Zivin & Tal Gross & Matthew J. Neidell, 2014. "Particulate Pollution and the Productivity of Pear Packers," NBER Working Papers 19944, National Bureau of Economic Research, Inc.
  2. Liu, Chu-An, 2013. "Distribution Theory of the Least Squares Averaging Estimator," MPRA Paper 54201, University Library of Munich, Germany.

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.