IDEAS home Printed from https://ideas.repec.org/p/zbw/iwhdps/22019.html
   My bibliography  Save this paper

An evaluation of early warning models for systemic banking crises: Does machine learning improve predictions?

Author

Listed:
  • Beutel, Johannes
  • List, Sophia
  • von Schweinitz, Gregor

Abstract

This paper compares the out-of-sample predictive performance of different early warning models for systemic banking crises using a sample of advanced economies covering the past 45 years. We compare a benchmark logit approach to several machine learning approaches recently proposed in the literature. We find that while machine learning methods often attain a very high in-sample fit, they are outperformed by the logit approach in recursive out-of-sample evaluations. This result is robust to the choice of performance measure, crisis definition, preference parameter, and sample length, as well as to using different sets of variables and data transformations. Thus, our paper suggests that further enhancements to machine learning early warning models are needed before they are able to offer a substantial value-added for predicting systemic banking crises. Conventional logit models appear to use the available information already fairly effciently, and would for instance have been able to predict the 2007/2008 financial crisis out-of-sample for many countries. In line with economic intuition, these models identify credit expansions, asset price booms and external imbalances as key predictors of systemic banking crises.

Suggested Citation

  • Beutel, Johannes & List, Sophia & von Schweinitz, Gregor, 2019. "An evaluation of early warning models for systemic banking crises: Does machine learning improve predictions?," IWH Discussion Papers 2/2019, Halle Institute for Economic Research (IWH).
  • Handle: RePEc:zbw:iwhdps:22019
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/191248/1/104687456X.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Duprey, Thibaut & Klaus, Benjamin & Peltonen, Tuomas, 2017. "Dating systemic financial stress episodes in the EU countries," Journal of Financial Stability, Elsevier, vol. 32(C), pages 30-56.
    2. Alessi, Lucia & Detken, Carsten, 2018. "Identifying excessive credit growth and leverage," Journal of Financial Stability, Elsevier, vol. 35(C), pages 215-225.
    3. McFadden, Daniel L., 1984. "Econometric analysis of qualitative response models," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 2, chapter 24, pages 1395-1457, Elsevier.
    4. Carmen M. Reinhart & Graciela L. Kaminsky, 1999. "The Twin Crises: The Causes of Banking and Balance-of-Payments Problems," American Economic Review, American Economic Association, vol. 89(3), pages 473-500, June.
    5. El-Shagi, M. & Knedlik, T. & von Schweinitz, G., 2013. "Predicting financial crises: The (statistical) significance of the signals approach," Journal of International Money and Finance, Elsevier, vol. 35(C), pages 76-103.
    6. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    7. Frankel, Jeffrey A & Rose, Andrew K, 1996. "Currency Crashes in Emerging Markets: Empirical Indicators," CEPR Discussion Papers 1349, C.E.P.R. Discussion Papers.
    8. Carmen M. Reinhart & Kenneth S. Rogoff, 2014. "This Time is Different: A Panoramic View of Eight Centuries of Financial Crises," Annals of Economics and Finance, Society for AEF, vol. 15(2), pages 215-268, November.
    9. Sarlin, Peter, 2013. "On policymakers’ loss functions and the evaluation of early warning systems," Economics Letters, Elsevier, vol. 119(1), pages 1-7.
    10. A. Berg & C. Pattillo, 1999. "What Caused the Asian Crises: An Early Warning System Approach," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 28(3), pages 285-334, November.
    11. Babecký, Jan & Havránek, Tomáš & Matějů, Jakub & Rusnák, Marek & Šmídková, Kateřina & Vašíček, Bořek, 2014. "Banking, debt, and currency crises in developed countries: Stylized facts and early warning indicators," Journal of Financial Stability, Elsevier, vol. 15(C), pages 1-17.
    12. Tobias Knedlik & Gregor Von Schweinitz, 2012. "Macroeconomic Imbalances as Indicators for Debt Crises in Europe," Journal of Common Market Studies, Wiley Blackwell, vol. 50(5), pages 726-745, September.
    13. Kaminsky, Graciela L. & Reinhart, Carmen M., 2000. "On crises, contagion, and confusion," Journal of International Economics, Elsevier, vol. 51(1), pages 145-168, June.
    14. Detken, Carsten & Weeken, Olaf & Alessi, Lucia & Bonfim, Diana & Boucinha, Miguel & Castro, Christian & Frontczak, Sebastian & Giordana, Gaston & Giese, Julia & Wildmann, Nadya & Kakes, Jan & Klaus, B, 2014. "Operationalising the countercyclical capital buffer: indicator selection, threshold identification and calibration options," ESRB Occasional Paper Series 5, European Systemic Risk Board.
    15. Reinhart, Carmen & Rogoff, Kenneth, 2009. "This Time It’s Different: Eight Centuries of Financial Folly-Preface," MPRA Paper 17451, University Library of Munich, Germany.
    16. Lo Duca, Marco & Koban, Anne & Basten, Marisa & Bengtsson, Elias & Klaus, Benjamin & Kusmierczyk, Piotr & Lang, Jan Hannes & Detken, Carsten & Peltonen, Tuomas A., 2017. "A new database for financial crises in European countries," Occasional Paper Series 194, European Central Bank.
    17. Carmen M. Reinhart & Kenneth S. Rogoff, 2009. "Varieties of Crises and Their Dates," Introductory Chapters, in: This Time Is Different: Eight Centuries of Financial Folly, Princeton University Press.
    18. Sarlin, Peter & von Schweinitz, Gregor, 2015. "Optimizing Policymakers' Loss Functions in Crisis Prediction: Before, Within or After?," IWH Discussion Papers 6/2015, Halle Institute for Economic Research (IWH).
    19. Reinhart, Carmen & Rogoff, Kenneth, 2009. "This Time It’s Different: Eight Centuries of Financial Folly-Chapter 1," MPRA Paper 17452, University Library of Munich, Germany.
    20. Fabian Valencia & Luc Laeven, 2012. "Systemic Banking Crises Database; An Update," IMF Working Papers 12/163, International Monetary Fund.
    21. Alessi, Lucia & Detken, Carsten, 2011. "Quasi real time early warning indicators for costly asset price boom/bust cycles: A role for global liquidity," European Journal of Political Economy, Elsevier, vol. 27(3), pages 520-533, September.
    22. Òscar Jordà & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 59(2), pages 340-378, June.
    23. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    24. Markus K. Brunnermeier, 2009. "Deciphering the Liquidity and Credit Crunch 2007-2008," Journal of Economic Perspectives, American Economic Association, vol. 23(1), pages 77-100, Winter.
    25. Makram El-Shagi & Axel Lindner & Gregor von Schweinitz, 2016. "Real Effective Exchange Rate Misalignment in the Euro Area: A Counterfactual Analysis," Review of International Economics, Wiley Blackwell, vol. 24(1), pages 37-66, February.
    26. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    27. Stijn Claessens, 2015. "An Overview of Macroprudential Policy Tools," Annual Review of Financial Economics, Annual Reviews, vol. 7(1), pages 397-422, December.
    28. Cerutti, Eugenio & Claessens, Stijn & Laeven, Luc, 2017. "The use and effectiveness of macroprudential policies: New evidence," Journal of Financial Stability, Elsevier, vol. 28(C), pages 203-224.
    29. repec:taf:quantf:v:17:y:2017:i:12:p:1933-1963 is not listed on IDEAS
    30. Raghuram G. Rajan, 2005. "Has financial development made the world riskier?," Proceedings - Economic Policy Symposium - Jackson Hole, Federal Reserve Bank of Kansas City, issue aug, pages 313-369.
    31. Angela Maddaloni & Jose-Luis Peydro, 2011. "Bank Risk-taking, Securitization, Supervision, and Low Interest Rates: Evidence from the Euro-area and the U.S. Lending Standards," Review of Financial Studies, Society for Financial Studies, vol. 24(6), pages 2121-2165.
    32. Guillermo A. Calvo, 1998. "Capital Flows and Capital-Market Crises: The Simple Economics of Sudden Stops," Journal of Applied Economics, Universidad del CEMA, vol. 1, pages 35-54, November.
    33. Frankel, Jeffrey A. & Rose, Andrew K., 1996. "Currency crashes in emerging markets: An empirical treatment," Journal of International Economics, Elsevier, vol. 41(3-4), pages 351-366, November.
    34. Mathias Drehmann & Claudio Borio & Kostas Tsatsaronis, 2012. "Characterising the financial cycle: don't lose sight of the medium term!," BIS Working Papers 380, Bank for International Settlements.
    35. Lutz Kilian, 1998. "Small-Sample Confidence Intervals For Impulse Response Functions," The Review of Economics and Statistics, MIT Press, vol. 80(2), pages 218-230, May.
    36. repec:mes:challe:v:25:y:1982:i:3:p:5-13 is not listed on IDEAS
    37. Frédéric Boissay & Fabrice Collard & Frank Smets, 2016. "Booms and Banking Crises," Journal of Political Economy, University of Chicago Press, vol. 124(2), pages 489-538.
    38. Tanaka, Katsuyuki & Kinkyo, Takuji & Hamori, Shigeyuki, 2016. "Random forests-based early warning system for bank failures," Economics Letters, Elsevier, vol. 148(C), pages 118-121.
    39. International Monetary Fund, 2011. "Macroprudential Policy; What Instruments and How to Use them? Lessons From Country Experiences," IMF Working Papers 11/238, International Monetary Fund.
    40. Carsten Detken & Olaf Weeken & Lucia Alessi & Diana Bonfim & Miguel M. Boucinha & Christian Castro & Sebastian Frontczak & Gaston Giordana & Julia Giese & Nadya Jahn & Jan Kakes & Benjamin Klaus & Jan, 2014. "Operationalising the countercyclical capital buffer: indicator selection, threshold identification and calibration options," ESRB Occasional Paper Series 05, European Systemic Risk Board.
    41. Mathias Drehmann & Mikael Juselius & Anton Korinek, 2017. "Accounting for debt service: the painful legacy of credit booms," BIS Working Papers 645, Bank for International Settlements.
    42. Mathias Drehmann & Claudio Borio & Kostas Tsatsaronis, 2011. "Anchoring Countercyclical Capital Buffers: The role of Credit Aggregates," International Journal of Central Banking, International Journal of Central Banking, vol. 7(4), pages 189-240, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Denis Shibitov & Mariam Mamedli, 2019. "The finer points of model comparison in machine learning: forecasting based on russian banks’ data," Bank of Russia Working Paper Series wps43, Bank of Russia.
    2. Sondermann, David & Zorell, Nico, 2019. "A macroeconomic vulnerability model for the euro area," Working Paper Series 2306, European Central Bank.

    More about this item

    Keywords

    early warning system; logit; machine learning; systemic banking crises;

    JEL classification:

    • C35 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Discrete Regression and Qualitative Choice Models; Discrete Regressors; Proportions
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • G01 - Financial Economics - - General - - - Financial Crises

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:iwhdps:22019. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (ZBW - Leibniz Information Centre for Economics). General contact details of provider: http://edirc.repec.org/data/iwhhhde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.