IDEAS home Printed from https://ideas.repec.org/a/eee/jbfina/v56y2015icp72-85.html
   My bibliography  Save this article

An empirical evaluation of the performance of binary classifiers in the prediction of credit ratings changes

Author

Listed:
  • Jones, Stewart
  • Johnstone, David
  • Wilson, Roy

Abstract

In this study, we examine the predictive performance of a wide class of binary classifiers using a large sample of international credit ratings changes from the period 1983–2013. Using a number of financial, market, corporate governance, macro-economic and other indicators as explanatory variables, we compare classifiers ranging from conventional techniques (such as logit/probit and LDA) to fully nonlinear classifiers, including neural networks, support vector machines and more recent statistical learning techniques such as generalised boosting, AdaBoost and random forests. We find that the newer classifiers significantly outperform all other classifiers on both the cross sectional and longitudinal test samples; and prove remarkably robust to different data structures and assumptions. Simple linear classifiers such as logit/probit and LDA are found nonetheless to predict quite accurately on the test samples, in some cases performing comparably well to more flexible model structures. We conclude that simpler classifiers can be viable alternatives to more sophisticated approaches, particularly if interpretability is an important objective of the modelling exercise. We also suggest effective ways to enhance the predictive performance of many of the binary classifiers examined in this study.

Suggested Citation

  • Jones, Stewart & Johnstone, David & Wilson, Roy, 2015. "An empirical evaluation of the performance of binary classifiers in the prediction of credit ratings changes," Journal of Banking & Finance, Elsevier, vol. 56(C), pages 72-85.
  • Handle: RePEc:eee:jbfina:v:56:y:2015:i:c:p:72-85
    DOI: 10.1016/j.jbankfin.2015.02.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378426615000333
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jbankfin.2015.02.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marshall E. Blume & Felix Lim & A. Craig MacKinlay, "undated". "The Declining Credit Quality of US Corporate Debt: Myth or Reality?," Rodney L. White Center for Financial Research Working Papers 3-98, Wharton School Rodney L. White Center for Financial Research.
    2. Hand, John R M & Holthausen, Robert W & Leftwich, Richard W, 1992. "The Effect of Bond Rating Agency Announcements on Bond and Stock Prices," Journal of Finance, American Finance Association, vol. 47(2), pages 733-752, June.
    3. Koopman, Siem Jan & Kräussl, Roman & Lucas, André & Monteiro, André B., 2009. "Credit cycles and macro fundamentals," Journal of Empirical Finance, Elsevier, vol. 16(1), pages 42-54, January.
    4. Joy, O. Maurice & Tollefson, John O., 1975. "On the Financial Applications of Discriminant Analysis," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 10(5), pages 723-739, December.
    5. Alex Frino & Stewart Jones & Jin Boon Wong, 2007. "Market behaviour around bankruptcy announcements: evidence from the Australian Stock Exchange," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 47(4), pages 713-730, December.
    6. Zhang, Guoqiang & Y. Hu, Michael & Eddy Patuwo, B. & C. Indro, Daniel, 1999. "Artificial neural networks in bankruptcy prediction: General framework and cross-validation analysis," European Journal of Operational Research, Elsevier, vol. 116(1), pages 16-32, July.
    7. David A. Hensher & Stewart Jones & William H. Greene, 2007. "An Error Component Logit Analysis of Corporate Bankruptcy and Insolvency Risk in Australia," The Economic Record, The Economic Society of Australia, vol. 83(260), pages 86-103, March.
    8. Shleifer, Andrei & Vishny, Robert W, 1997. "A Survey of Corporate Governance," Journal of Finance, American Finance Association, vol. 52(2), pages 737-783, June.
    9. Nickell, Pamela & Perraudin, William & Varotto, Simone, 2000. "Stability of rating transitions," Journal of Banking & Finance, Elsevier, vol. 24(1-2), pages 203-227, January.
    10. repec:hrv:faseco:30728046 is not listed on IDEAS
    11. B Baesens & T Van Gestel & S Viaene & M Stepanova & J Suykens & J Vanthienen, 2003. "Benchmarking state-of-the-art classification algorithms for credit scoring," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 54(6), pages 627-635, June.
    12. Jones,Stewart & Hensher,David A. (ed.), 2008. "Advances in Credit Risk Modelling and Corporate Bankruptcy Prediction," Cambridge Books, Cambridge University Press, number 9780521689540.
    13. Altman, Edward I. & Rijken, Herbert A., 2004. "How rating agencies achieve rating stability," Journal of Banking & Finance, Elsevier, vol. 28(11), pages 2679-2714, November.
    14. Kar Yan Tam & Melody Y. Kiang, 1992. "Managerial Applications of Neural Networks: The Case of Bank Failure Predictions," Management Science, INFORMS, vol. 38(7), pages 926-947, July.
    15. Ilia D. Dichev & Joseph D. Piotroski, 2001. "The Long‐Run Stock Returns Following Bond Ratings Changes," Journal of Finance, American Finance Association, vol. 56(1), pages 173-203, February.
    16. Michael Doumpos & Constantin Zopounidis, 2007. "Model combination for credit risk assessment: A stacked generalization approach," Annals of Operations Research, Springer, vol. 151(1), pages 289-306, April.
    17. Altman, Edward I. & Marco, Giancarlo & Varetto, Franco, 1994. "Corporate distress diagnosis: Comparisons using linear discriminant analysis and neural networks (the Italian experience)," Journal of Banking & Finance, Elsevier, vol. 18(3), pages 505-529, May.
    18. Bangia, Anil & Diebold, Francis X. & Kronimus, Andre & Schagen, Christian & Schuermann, Til, 2002. "Ratings migration and the business cycle, with application to credit portfolio stress testing," Journal of Banking & Finance, Elsevier, vol. 26(2-3), pages 445-474, March.
    19. Figlewski, Stephen & Frydman, Halina & Liang, Weijian, 2012. "Modeling the effect of macroeconomic factors on corporate default and credit rating transitions," International Review of Economics & Finance, Elsevier, vol. 21(1), pages 87-105.
    20. Marshall E. Blume & Felix Lim & A. Craig Mackinlay, 1998. "The Declining Credit Quality of U.S. Corporate Debt: Myth or Reality?," Journal of Finance, American Finance Association, vol. 53(4), pages 1389-1413, August.
    21. Altman, Edward I. & Haldeman, Robert G. & Narayanan, P., 1977. "ZETATM analysis A new model to identify bankruptcy risk of corporations," Journal of Banking & Finance, Elsevier, vol. 1(1), pages 29-54, June.
    22. Amato, Jeffery D. & Furfine, Craig H., 2004. "Are credit ratings procyclical?," Journal of Banking & Finance, Elsevier, vol. 28(11), pages 2641-2677, November.
    23. Shumway, Tyler, 2001. "Forecasting Bankruptcy More Accurately: A Simple Hazard Model," The Journal of Business, University of Chicago Press, vol. 74(1), pages 101-124, January.
    24. Koopman, Siem Jan & Lucas, André & Schwaab, Bernd, 2011. "Modeling frailty-correlated defaults using many macroeconomic covariates," Journal of Econometrics, Elsevier, vol. 162(2), pages 312-325, June.
    25. Jones,Stewart & Hensher,David A. (ed.), 2008. "Advances in Credit Risk Modelling and Corporate Bankruptcy Prediction," Cambridge Books, Cambridge University Press, number 9780521869287.
    26. Zmijewski, Me, 1984. "Methodological Issues Related To The Estimation Of Financial Distress Prediction Models," Journal of Accounting Research, Wiley Blackwell, vol. 22, pages 59-82.
    27. Ashbaugh-Skaife, Hollis & Collins, Daniel W. & LaFond, Ryan, 2006. "The effects of corporate governance on firms' credit ratings," Journal of Accounting and Economics, Elsevier, vol. 42(1-2), pages 203-243, October.
    28. Lando, David & Skodeberg, Torben M., 2002. "Analyzing rating transitions and rating drift with continuous observations," Journal of Banking & Finance, Elsevier, vol. 26(2-3), pages 423-444, March.
    29. Marshall E. Blume & Felix Lim & A. Craig MacKinlay, "undated". "The Declining Credit Quality of US Corporate Debt: Myth or Reality?," Rodney L. White Center for Financial Research Working Papers 03-98, Wharton School Rodney L. White Center for Financial Research.
    30. Christensen, Jens H.E. & Hansen, Ernst & Lando, David, 2004. "Confidence sets for continuous-time rating transition probabilities," Journal of Banking & Finance, Elsevier, vol. 28(11), pages 2575-2602, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stewart Jones, 2017. "Corporate bankruptcy prediction: a high dimensional analysis," Review of Accounting Studies, Springer, vol. 22(3), pages 1366-1422, September.
    2. Van Laere, Elisabeth & Baesens, Bart, 2010. "The development of a simple and intuitive rating system under Solvency II," Insurance: Mathematics and Economics, Elsevier, vol. 46(3), pages 500-510, June.
    3. Jones, Stewart & Wang, Tim, 2019. "Predicting private company failure: A multi-class analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 161-188.
    4. Rubina Shaheen & Attiya Yasmin Javid, 2014. "Effect of Credit Rating on Firm Performance and Stock Return; Evidence form KSE Listed Firms," PIDE-Working Papers 2014:104, Pakistan Institute of Development Economics.
    5. Kerem Tuzcuoglu, 2019. "Composite Likelihood Estimation of an Autoregressive Panel Probit Model with Random Effects," Staff Working Papers 19-16, Bank of Canada.
    6. Xing, Haipeng & Sun, Ning & Chen, Ying, 2012. "Credit rating dynamics in the presence of unknown structural breaks," Journal of Banking & Finance, Elsevier, vol. 36(1), pages 78-89.
    7. Mizen, Paul & Tsoukas, Serafeim, 2012. "Forecasting US bond default ratings allowing for previous and initial state dependence in an ordered probit model," International Journal of Forecasting, Elsevier, vol. 28(1), pages 273-287.
    8. Mohammad Shamsu Uddin & Guotai Chi & Mazin A. M. Al Janabi & Tabassum Habib & Kunpeng Yuan, 2022. "Modeling credit risk with a multi‐stage hybrid model: An alternative statistical approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(7), pages 1386-1415, November.
    9. Shen, Chung-Hua & Huang, Yu-Li & Hasan, Iftekhar, 2012. "Asymmetric benchmarking in bank credit rating," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 22(1), pages 171-193.
    10. Demirtas, K. Ozgur & Rodgers Cornaggia, Kimberly, 2013. "Initial credit ratings and earnings management," Review of Financial Economics, Elsevier, vol. 22(4), pages 135-145.
    11. Gerald J. Lobo & Luc Paugam & Hervé Stolowy & Pierre Astolfi, 2017. "The Effect of Business and Financial Market Cycles on Credit Ratings: Evidence from the Last Two Decades," Abacus, Accounting Foundation, University of Sydney, vol. 53(1), pages 59-93, March.
    12. Koresh Galil, 2005. "Ratings as Predictors of Default in the Long Term:an Empirical Investigation," Working Papers 0505, Ben-Gurion University of the Negev, Department of Economics.
    13. fernández, María t. Tascón & gutiérrez, Francisco J. Castaño, 2012. "Variables y Modelos Para La Identificación y Predicción Del Fracaso Empresarial: Revisión de La Investigación Empírica Reciente," Revista de Contabilidad - Spanish Accounting Review, Elsevier, vol. 15(1), pages 7-58.
    14. Maurice Peat & Stewart Jones, 2012. "Using Neural Nets To Combine Information Sets In Corporate Bankruptcy Prediction," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 19(2), pages 90-101, April.
    15. Shen, Chung-Hua & Huang, Yu-Li & Hasan, Iftekhar, 2012. "Asymmetric benchmarking in bank credit rating," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 22(1), pages 171-193.
    16. Eleimon Gonis & Salima Paul & Jon Tucker, 2012. "Rating or no rating? That is the question: an empirical examination of UK companies," The European Journal of Finance, Taylor & Francis Journals, vol. 18(8), pages 709-735, September.
    17. Güttler, André & Raupach, Peter, 2008. "The impact of downward rating momentum on credit portfolio risk," Discussion Paper Series 2: Banking and Financial Studies 2008,16, Deutsche Bundesbank.
    18. repec:zbw:bofrdp:2012_013 is not listed on IDEAS
    19. Trueck, Stefan & Rachev, Svetlozar T., 2008. "Rating Based Modeling of Credit Risk," Elsevier Monographs, Elsevier, edition 1, number 9780123736833.
    20. Figlewski, Stephen & Frydman, Halina & Liang, Weijian, 2012. "Modeling the effect of macroeconomic factors on corporate default and credit rating transitions," International Review of Economics & Finance, Elsevier, vol. 21(1), pages 87-105.
    21. Kraft, Pepa & Xie, Yuan & Zhou, Ling, 2020. "The intraday timing of rating changes," Journal of Corporate Finance, Elsevier, vol. 60(C).

    More about this item

    Keywords

    Credit ratings changes; Prediction; Binary classifiers; Statistical learning;
    All these keywords.

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • M4 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jbfina:v:56:y:2015:i:c:p:72-85. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.elsevier.com/locate/jbf .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jbf .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.