IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v178y2014ip1p194-206.html
   My bibliography  Save this article

Robustness checks and robustness tests in applied economics

Author

Listed:
  • Lu, Xun
  • White, Halbert

Abstract

A common exercise in empirical studies is a “robustness check”, where the researcher examines how certain “core” regression coefficient estimates behave when the regression specification is modified by adding or removing regressors. If the coefficients are plausible and robust, this is commonly interpreted as evidence of structural validity. Here, we study when and how one can infer structural validity from coefficient robustness and plausibility. As we show, there are numerous pitfalls, as commonly implemented robustness checks give neither necessary nor sufficient evidence for structural validity. Indeed, if not conducted properly, robustness checks can be completely uninformative or entirely misleading. We discuss how critical and non-critical core variables can be properly specified and how non-core variables for the comparison regression can be chosen to ensure that robustness checks are indeed structurally informative. We provide a straightforward new Hausman (1978) type test of robustness for the critical core coefficients, additional diagnostics that can help explain why robustness test rejection occurs, and a new estimator, the Feasible Optimally combined GLS (FOGLeSs) estimator, that makes relatively efficient use of the robustness check regressions. A new procedure for Matlab, testrob, embodies these methods.

Suggested Citation

  • Lu, Xun & White, Halbert, 2014. "Robustness checks and robustness tests in applied economics," Journal of Econometrics, Elsevier, vol. 178(P1), pages 194-206.
  • Handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206
    DOI: 10.1016/j.jeconom.2013.08.016
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304407613001668
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
    2. White, Halbert, 2006. "Time-series estimation of the effects of natural experiments," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 527-566.
    3. Andrews, Donald W K, 1994. "Asymptotics for Semiparametric Econometric Models via Stochastic Equicontinuity," Econometrica, Econometric Society, vol. 62(1), pages 43-72, January.
    4. Alexandre Mas & Enrico Moretti, 2009. "Peers at Work," American Economic Review, American Economic Association, vol. 99(1), pages 112-145, March.
    5. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
    6. Miguel Urquiola & Eric Verhoogen, 2009. "Class-Size Caps, Sorting, and the Regression-Discontinuity Design," American Economic Review, American Economic Association, vol. 99(1), pages 179-215, March.
    7. Halbert White & Xun Lu, 2010. "Granger Causality and Dynamic Structural Systems," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 8(2), pages 193-243, spring.
    8. Jinyong Hahn, 2004. "Functional Restriction and Efficiency in Causal Inference," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 73-76, February.
    9. William Adams & Liran Einav & Jonathan Levin, 2009. "Liquidity Constraints and Imperfect Information in Subprime Lending," American Economic Review, American Economic Association, vol. 99(1), pages 49-84, March.
    10. Laura Alfaro & Andrew Charlton, 2009. "Intra-industry Foreign Direct Investment," American Economic Review, American Economic Association, vol. 99(5), pages 2096-2119, December.
    11. Jean Boivin & Marc P. Giannoni & Ilian Mihov, 2009. "Sticky Prices and Monetary Policy: Evidence from Disaggregated US Data," American Economic Review, American Economic Association, vol. 99(1), pages 350-384, March.
    12. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-1277, September.
    13. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
    14. Raj Chetty & Adam Looney & Kory Kroft, 2009. "Salience and Taxation: Theory and Evidence," American Economic Review, American Economic Association, vol. 99(4), pages 1145-1177, September.
    15. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    16. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
    17. Michael D. Makowsky & Thomas Stratmann, 2009. "Political Economy at Any Speed: What Determines Traffic Citations?," American Economic Review, American Economic Association, vol. 99(1), pages 509-527, March.
    18. Antonio Spilimbergo, 2009. "Democracy and Foreign Education," American Economic Review, American Economic Association, vol. 99(1), pages 528-543, March.
    19. Felix Oberholzer-Gee & Joel Waldfogel, 2009. "Media Markets and Localism: Does Local News en Español Boost Hispanic Voter Turnout?," American Economic Review, American Economic Association, vol. 99(5), pages 2120-2128, December.
    20. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    21. Karim Chalak & Halbert White, 2008. "Causality, Conditional Independence, and Graphical Separation in Settable Systems," Boston College Working Papers in Economics 689, Boston College Department of Economics, revised 04 Jul 2010.
    22. Robinson, P M, 1987. "Asymptotically Efficient Estimation in the Presence of Heteroskedasticity of Unknown Form," Econometrica, Econometric Society, vol. 55(4), pages 875-891, July.
    23. Francisco Pérez-González, 2006. "Inherited Control and Firm Performance," American Economic Review, American Economic Association, vol. 96(5), pages 1559-1588, December.
    24. Gallant, A. Ronald, 1982. "Unbiased determination of production technologies," Journal of Econometrics, Elsevier, vol. 20(2), pages 285-323, November.
    25. Igal Hendel & Aviv Nevo & François Ortalo-Magné, 2009. "The Relative Performance of Real Estate Marketing Platforms: MLS versus FSBOMadison.com," American Economic Review, American Economic Association, vol. 99(5), pages 1878-1898, December.
    26. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    27. Karim Chalak & Halbert White, 2011. "Viewpoint: An extended class of instrumental variables for the estimation of causal effects," Canadian Journal of Economics, Canadian Economics Association, vol. 44(1), pages 1-51, February.
    28. Hausman, Jerry, 2015. "Specification tests in econometrics," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 38(2), pages 112-134.
    29. Dahl, Christian M. & Gonzalez-Rivera, Gloria, 2003. "Testing for neglected nonlinearity in regression models based on the theory of random fields," Journal of Econometrics, Elsevier, vol. 114(1), pages 141-164, May.
    30. Carlos Dobkin & Nancy Nicosia, 2009. "The War on Drugs: Methamphetamine, Public Health, and Crime," American Economic Review, American Economic Association, vol. 99(1), pages 324-349, March.
    31. White, Halbert & Chalak, Karim, 2010. "Testing a conditional form of exogeneity," Economics Letters, Elsevier, vol. 109(2), pages 88-90, November.
    32. Hongbin Cai & Yuyu Chen & Hanming Fang, 2009. "Observational Learning: Evidence from a Randomized Natural Field Experiment," American Economic Review, American Economic Association, vol. 99(3), pages 864-882, June.
    33. Nathan H. Miller, 2009. "Strategic Leniency and Cartel Enforcement," American Economic Review, American Economic Association, vol. 99(3), pages 750-768, June.
    34. Hamilton, James D, 2001. "A Parametric Approach to Flexible Nonlinear Inference," Econometrica, Econometric Society, vol. 69(3), pages 537-573, May.
    35. Lee, Tae-Hwy & White, Halbert & Granger, Clive W. J., 1993. "Testing for neglected nonlinearity in time series models : A comparison of neural network methods and alternative tests," Journal of Econometrics, Elsevier, vol. 56(3), pages 269-290, April.
    36. Manuela Angelucci & Giacomo De Giorgi, 2009. "Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?," American Economic Review, American Economic Association, vol. 99(1), pages 486-508, March.
    37. John G. Matsusaka, 2009. "Direct Democracy and Public Employees," American Economic Review, American Economic Association, vol. 99(5), pages 2227-2246, December.
    38. Shaun P. Hargreaves Heap & Daniel John Zizzo, 2009. "The Value of Groups," American Economic Review, American Economic Association, vol. 99(1), pages 295-323, March.
    39. Griliches, Zvi, 1977. "Estimating the Returns to Schooling: Some Econometric Problems," Econometrica, Econometric Society, vol. 45(1), pages 1-22, January.
    40. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    41. Halbert White & Xun Lu, 2011. "Causal Diagrams for Treatment Effect Estimation with Application to Efficient Covariate Selection," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1453-1459, November.
    42. Yan Chen & Sherry Xin Li, 2009. "Group Identity and Social Preferences," American Economic Review, American Economic Association, vol. 99(1), pages 431-457, March.
    43. Su, Liangjun & White, Halbert, 2010. "Testing Structural Change In Partially Linear Models," Econometric Theory, Cambridge University Press, vol. 26(06), pages 1761-1806, December.
    44. Clare Leaver, 2009. "Bureaucratic Minimal Squawk Behavior: Theory and Evidence from Regulatory Agencies," American Economic Review, American Economic Association, vol. 99(3), pages 572-607, June.
    45. Clemens Sialm, 2009. "Tax Changes and Asset Pricing," American Economic Review, American Economic Association, vol. 99(4), pages 1356-1383, September.
    46. Silke Januszewski Forbes & Mara Lederman, 2009. "Adaptation and Vertical Integration in the Airline Industry," American Economic Review, American Economic Association, vol. 99(5), pages 1831-1849, December.
    47. White, Halbert, 2006. "Approximate Nonlinear Forecasting Methods," Handbook of Economic Forecasting, Elsevier.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Masayoshi Hayashi & Wataru Yamamoto, 2017. "Information sharing, neighborhood demarcation, and yardstick competition: an empirical analysis of intergovernmental expenditure interaction in Japan," International Tax and Public Finance, Springer;International Institute of Public Finance, vol. 24(1), pages 134-163, February.
    2. Fuje,Habtamu Neda & Tandon,Prateek, 2015. "When do in-service teacher training and books improve student achievement ? experimental evidence from Mongolia," Policy Research Working Paper Series 7485, The World Bank.
    3. Tom Chang & Joshua Graff Zivin & Tal Gross & Matthew Neidell, 2016. "Particulate Pollution and the Productivity of Pear Packers," American Economic Journal: Economic Policy, American Economic Association, vol. 8(3), pages 141-169, August.
    4. Dimitrios Bakas & Georgios Chortareas & Georgios Magkonis, 2017. "Volatility and Growth: A not so straightforward relationship," Working Paper series 17-12, Rimini Centre for Economic Analysis.
    5. Liu, Chu-An, 2015. "Distribution theory of the least squares averaging estimator," Journal of Econometrics, Elsevier, vol. 186(1), pages 142-159.
    6. repec:eee:intfin:v:48:y:2017:i:c:p:1-24 is not listed on IDEAS
    7. repec:eee:jfpoli:v:74:y:2018:i:c:p:1-8 is not listed on IDEAS
    8. repec:voj:journl:v:63:y:2016:i:2:p:231-258 is not listed on IDEAS
    9. repec:eee:enepol:v:114:y:2018:i:c:p:234-244 is not listed on IDEAS

    More about this item

    Keywords

    Robustness; Causal effect; Conditional exogeneity; Specification test; Combined estimator;

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:178:y:2014:i:p1:p:194-206. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.