A New Goodness-of-Fit Test for Event Forecasting and Its Application to Credit Defaults
We develop a new goodness-of-fit test for validating the performance of probability forecasts. Our test statistic is particularly powerful under sparseness and dependence in the observed data. To build our test statistic, we start from a formal definition of calibrated forecasts, which we operationalize by introducing two components. The first component tests the level of the estimated probabilities; the second validates the shape, measuring the differentiation between high and low probability events. After constructing test statistics for both level and shape, we provide a global goodness-of-fit statistic, which is asymptotically \chi 2 distributed. In a simulation exercise, we find that our approach is correctly sized and more powerful than alternative statistics. In particular, our shape statistic is significantly more powerful than the Kolmogorov-Smirnov test. Under independence, our global test has significantly greater power than the popular Hosmer-Lemeshow's \chi 2 test. Moreover, even under dependence, our global test remains correctly sized and consistent. As a timely and important empirical application of our method, we study the validation of a forecasting model for credit default events. This paper was accepted by Wei Xiong, finance.
Volume (Year): 57 (2011)
Issue (Month): 3 (March)
|Contact details of provider:|| Postal: 7240 Parkway Drive, Suite 300, Hanover, MD 21076 USA|
Web page: http://www.informs.org/
More information through EDIRC
References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Donald W.K. Andrews, 2004.
"Cross-section Regression with Common Shocks,"
Yale School of Management Working Papers
ysm401, Yale School of Management.
- White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
- Han Bleichrodt & Jose Luis Pinto, 2000. "A Parameter-Free Elicitation of the Probability Weighting Function in Medical Decision Analysis," Management Science, INFORMS, vol. 46(11), pages 1485-1496, November.
- G. Noether, 1963. "Note on the kolmogorov statistic in the discrete case," Metrika- International Journal for Theoretical and Applied Statistics, Springer, vol. 7(1), pages 115-116, December.
- Gordy, Michael B., 2003.
"A risk-factor model foundation for ratings-based bank capital rules,"
Journal of Financial Intermediation,
Elsevier, vol. 12(3), pages 199-232, July.
- Michael B. Gordy, 2002. "A risk-factor model foundation for ratings-based bank capital rules," Finance and Economics Discussion Series 2002-55, Board of Governors of the Federal Reserve System (U.S.).
- Sanjiv R. Das & Darrell Duffie & Nikunj Kapadia & Leandro Saita, 2007.
"Common Failings: How Corporate Defaults Are Correlated,"
Journal of Finance,
American Finance Association, vol. 62(1), pages 93-117, 02.
- Sanjiv Das & Darrell Duffie & Nikunj Kapadia & Leandro Saita, 2006. "Common Failings: How Corporate Defaults are Correlated," NBER Working Papers 11961, National Bureau of Economic Research, Inc.
- Kate Krause & Lise Vesterlund & William Harbaugh, 2002.
"Risk attitudes of children and adults: Choices over small and large probability gains and losses,"
Artefactual Field Experiments
00055, The Field Experiments Website.
- William Harbaugh & Kate Krause & Lise Vesterlund, 2002. "Risk Attitudes of Children and Adults: Choices Over Small and Large Probability Gains and Losses," Experimental Economics, Springer, vol. 5(1), pages 53-84, June.
- William T. Harbaugh & Kate Krause & Lise Vesterlund, 1999. "Risk attitudes of children and adults: choices over small and large probability gains and losses," University of Oregon Economics Department Working Papers 1999-2, University of Oregon Economics Department.
- Wei Pan, 2002. "Goodness-of-fit Tests for GEE with Correlated Binary Data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 29(1), pages 101-110.
- Hanson, Samuel & Schuermann, Til, 2006. "Confidence intervals for probabilities of default," Journal of Banking & Finance, Elsevier, vol. 30(8), pages 2281-2301, August.
- Michael B. Gordy, 1998.
"A comparative anatomy of credit risk models,"
Finance and Economics Discussion Series
1998-47, Board of Governors of the Federal Reserve System (U.S.).
- Archer, Kellie J. & Lemeshow, Stanley & Hosmer, David W., 2007. "Goodness-of-fit tests for logistic regression models when data are collected using a complex sampling design," Computational Statistics & Data Analysis, Elsevier, vol. 51(9), pages 4450-4464, May.
- Cubitt, Robin P & Starmer, Chris & Sugden, Robert, 1998. "Dynamic Choice and the Common Ratio Effect: An Experimental Investigation," Economic Journal, Royal Economic Society, vol. 108(450), pages 1362-80, September.
- Orme, Christopher, 1988. "The Calculation of the Information Matrix Test for Binary Data Models," The Manchester School of Economic & Social Studies, University of Manchester, vol. 56(4), pages 370-76, December.
- Blochlinger, Andreas & Leippold, Markus, 2006. "Economic benefit of powerful credit scoring," Journal of Banking & Finance, Elsevier, vol. 30(3), pages 851-873, March.
When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:57:y:2011:i:3:p:487-505. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mirko Janc)
If references are entirely missing, you can add them using this form.