IDEAS home Printed from https://ideas.repec.org/p/ecb/ecbwps/20172025.html
   My bibliography  Save this paper

Optimizing policymakers' loss functions in crisis prediction: before, within or after?

Author

Listed:
  • Sarlin, Peter
  • von Schweinitz, Gregor

Abstract

Early-warning models most commonly optimize signaling thresholds on crisis probabilities. The ex-post threshold optimization is based upon a loss function accounting for preferences between forecast errors, but comes with two crucial drawbacks: unstable thresholds in recursive estimations and an in-sample overfit at the expense of out-of-sample performance. We propose two alternatives for threshold setting: (i) including preferences in the estimation itself and (ii) setting thresholds ex-ante according to preferences only. Given probabilistic model output, it is intuitive that a decision rule is independent of the data or model specification, as thresholds on probabilities represent a willingness to issue a false alarm vis- JEL Classification: C35, C53, G01

Suggested Citation

  • Sarlin, Peter & von Schweinitz, Gregor, 2017. "Optimizing policymakers' loss functions in crisis prediction: before, within or after?," Working Paper Series 2025, European Central Bank.
    • Handle: RePEc:ecb:ecbwps:20172025
    as

    Download full text from publisher

    File URL: https://www.ecb.europa.eu//pub/pdf/scpwps/ecbwp2025.en.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 38(2), pages 257-279.
    2. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    3. Frankel, Jeffrey A. & Rose, Andrew K., 1996. "Currency Crashes in Emerging Markets: Empirical Indicators," Center for International and Development Economics Research (CIDER) Working Papers 233424, University of California-Berkeley, Department of Economics.
    4. Sarlin, Peter, 2013. "On policymakers’ loss functions and the evaluation of early warning systems," Economics Letters, Elsevier, vol. 119(1), pages 1-7.
    5. Tobias Knedlik & Gregor Von Schweinitz, 2012. "Macroeconomic Imbalances as Indicators for Debt Crises in Europe," Journal of Common Market Studies, Wiley Blackwell, vol. 50(5), pages 726-745, September.
    6. Carmen M. Reinhart & Graciela L. Kaminsky, 1999. "The Twin Crises: The Causes of Banking and Balance-of-Payments Problems," American Economic Review, American Economic Association, vol. 89(3), pages 473-500, June.
    7. Alessi, Lucia & Detken, Carsten, 2011. "Quasi real time early warning indicators for costly asset price boom/bust cycles: A role for global liquidity," European Journal of Political Economy, Elsevier, vol. 27(3), pages 520-533, September.
    8. Davis, E. Philip & Karim, Dilruba, 2008. "Comparing early warning systems for banking crises," Journal of Financial Stability, Elsevier, vol. 4(2), pages 89-120, June.
    9. King, Gary & Zeng, Langche, 2001. "Logistic Regression in Rare Events Data," Political Analysis, Cambridge University Press, vol. 9(2), pages 137-163, January.
    10. Russell Davidson & James MacKinnon, 2000. "Bootstrap tests: how many bootstraps?," Econometric Reviews, Taylor & Francis Journals, vol. 19(1), pages 55-68.
    11. Bussiere, Matthieu & Fratzscher, Marcel, 2008. "Low probability, high impact: Policy making and extreme events," Journal of Policy Modeling, Elsevier, vol. 30(1), pages 111-121.
    12. Demirguc, Asli & Detragiache, Enrica, 2000. "Monitoring Banking Sector Fragility: A Multivariate Logit Approach," The World Bank Economic Review, World Bank, vol. 14(2), pages 287-307, May.
    13. Betz, Frank & Oprică, Silviu & Peltonen, Tuomas A. & Sarlin, Peter, 2014. "Predicting distress in European banks," Journal of Banking & Finance, Elsevier, vol. 45(C), pages 225-241.
    14. Kumar, Mohan & Moorthy, Uma & Perraudin, William, 2003. "Predicting emerging market currency crashes," Journal of Empirical Finance, Elsevier, vol. 10(4), pages 427-454, September.
    15. El-Shagi, M. & Knedlik, T. & von Schweinitz, G., 2013. "Predicting financial crises: The (statistical) significance of the signals approach," Journal of International Money and Finance, Elsevier, vol. 35(C), pages 76-103.
    16. A. Berg & C. Pattillo, 1999. "What Caused the Asian Crises: An Early Warning System Approach," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 28(3), pages 285-334, November.
    17. repec:zbw:bofrdp:2015_006 is not listed on IDEAS
    18. Fuertes, Ana-Maria & Kalotychou, Elena, 2007. "Optimal design of early warning systems for sovereign debt crises," International Journal of Forecasting, Elsevier, vol. 23(1), pages 85-100.
    19. Frankel, Jeffrey A. & Rose, Andrew K., 1996. "Currency crashes in emerging markets: An empirical treatment," Journal of International Economics, Elsevier, vol. 41(3-4), pages 351-366, November.
    20. Bussiere, Matthieu & Fratzscher, Marcel, 2006. "Towards a new early warning system of financial crises," Journal of International Money and Finance, Elsevier, vol. 25(6), pages 953-973, October.
    21. Manski, Charles F & Lerman, Steven R, 1977. "The Estimation of Choice Probabilities from Choice Based Samples," Econometrica, Econometric Society, vol. 45(8), pages 1977-1988, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Beutel, Johannes & List, Sophia & von Schweinitz, Gregor, 2018. "An evaluation of early warning models for systemic banking crises: Does machine learning improve predictions?," Discussion Papers 48/2018, Deutsche Bundesbank.
    2. Donato Ceci & Andrea Silvestrini, 2023. "Nowcasting the state of the Italian economy: The role of financial markets," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1569-1593, November.
    3. Makram El‐Shagi & Gregor von Schweinitz, 2022. "Why they keep missing: An empirical investigation of sovereign bond ratings and their timing," Scottish Journal of Political Economy, Scottish Economic Society, vol. 69(2), pages 186-224, May.
    4. El-Shagi, Makram, 2017. "Dealing with small sample bias in post-crisis samples," Economic Modelling, Elsevier, vol. 65(C), pages 1-8.
    5. Beutel, Johannes & List, Sophia & von Schweinitz, Gregor, 2019. "Does machine learning help us predict banking crises?," Journal of Financial Stability, Elsevier, vol. 45(C).
    6. Quentin Bro de Comères, 2022. "Predicting European Banks Distress Events: Do Financial Information Producers Matter?," Working Papers hal-03752678, HAL.
    7. Casabianca, Elizabeth Jane & Catalano, Michele & Forni, Lorenzo & Giarda, Elena & Passeri, Simone, 2022. "A machine learning approach to rank the determinants of banking crises over time and across countries," Journal of International Money and Finance, Elsevier, vol. 129(C).
    8. Huynh, Tran & Uebelmesser, Silke, 2024. "Early warning models for systemic banking crises: Can political indicators improve prediction?," European Journal of Political Economy, Elsevier, vol. 81(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. von Schweinitz, Gregor & Sarlin, Peter, 2015. "Signaling Crises: How to Get Good Out-of-Sample Performance Out of the Early Warning System," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 112964, Verein für Socialpolitik / German Economic Association.
    2. Beutel, Johannes & List, Sophia & von Schweinitz, Gregor, 2019. "Does machine learning help us predict banking crises?," Journal of Financial Stability, Elsevier, vol. 45(C).
    3. Betz, Frank & Oprică, Silviu & Peltonen, Tuomas A. & Sarlin, Peter, 2014. "Predicting distress in European banks," Journal of Banking & Finance, Elsevier, vol. 45(C), pages 225-241.
    4. Beutel, Johannes & List, Sophia & von Schweinitz, Gregor, 2018. "An evaluation of early warning models for systemic banking crises: Does machine learning improve predictions?," Discussion Papers 48/2018, Deutsche Bundesbank.
    5. Tuomas Antero Peltonen & Michela Rancan & Peter Sarlin, 2019. "Interconnectedness of the banking sector as a vulnerability to crises," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 24(2), pages 963-990, April.
    6. Lanbiao Liu & Chen Chen & Bo Wang, 2022. "Predicting financial crises with machine learning methods," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(5), pages 871-910, August.
    7. Lo Duca, Marco & Koban, Anne & Basten, Marisa & Bengtsson, Elias & Klaus, Benjamin & Kusmierczyk, Piotr & Lang, Jan Hannes & Detken, Carsten & Peltonen, Tuomas, 2017. "A new database for financial crises in European countries," ESRB Occasional Paper Series 13, European Systemic Risk Board.
    8. El-Shagi, M. & Knedlik, T. & von Schweinitz, G., 2013. "Predicting financial crises: The (statistical) significance of the signals approach," Journal of International Money and Finance, Elsevier, vol. 35(C), pages 76-103.
    9. Oet, Mikhail V. & Gramlich, Dieter & Sarlin, Peter, 2016. "Evaluating measures of adverse financial conditions," Journal of Financial Stability, Elsevier, vol. 27(C), pages 234-249.
    10. Markus Behn & Carsten Detken & Tuomas Peltonen & Willem Schudel, 2017. "Predicting Vulnerabilities in the EU Banking Sector: The Role of Global and Domestic Factors," International Journal of Central Banking, International Journal of Central Banking, vol. 13(4), pages 147-189, December.
    11. Markus Holopainen & Peter Sarlin, 2015. "Toward robust early-warning models: A horse race, ensembles and model uncertainty," Papers 1501.04682, arXiv.org, revised Apr 2016.
    12. Sondermann, David & Zorell, Nico, 2019. "A macroeconomic vulnerability model for the euro area," Working Paper Series 2306, European Central Bank.
    13. Kauko, Karlo, 2014. "How to foresee banking crises? A survey of the empirical literature," Economic Systems, Elsevier, vol. 38(3), pages 289-308.
    14. Alessi, Lucia & Detken, Carsten, 2018. "Identifying excessive credit growth and leverage," Journal of Financial Stability, Elsevier, vol. 35(C), pages 215-225.
    15. Knedlik, Tobias, 2014. "The impact of preferences on early warning systems — The case of the European Commission's Scoreboard," European Journal of Political Economy, Elsevier, vol. 34(C), pages 157-166.
    16. Quentin Bro de Comères, 2022. "Predicting European Banks Distress Events: Do Financial Information Producers Matter?," Working Papers hal-03752678, HAL.
    17. Sarlin, Peter, 2013. "On policymakers’ loss functions and the evaluation of early warning systems," Economics Letters, Elsevier, vol. 119(1), pages 1-7.
    18. Lainà, Patrizio & Nyholm, Juho & Sarlin, Peter, 2015. "Leading indicators of systemic banking crises: Finland in a panel of EU countries," Review of Financial Economics, Elsevier, vol. 24(C), pages 18-35.
    19. Detken, Carsten & Peltonen, Tuomas A. & Schudel, Willem & Behn, Markus, 2013. "Setting countercyclical capital buffers based on early warning models: would it work?," Working Paper Series 1604, European Central Bank.
    20. Patrizio Lainà & Juho Nyholm & Peter Sarlin, 2015. "Leading indicators of systemic banking crises: Finland in a panel of EU countries," Review of Financial Economics, John Wiley & Sons, vol. 24(1), pages 18-35, January.

    More about this item

    Keywords

    early-warning models; loss functions; predictive performance; threshold setting;
    All these keywords.

    JEL classification:

    • C35 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Discrete Regression and Qualitative Choice Models; Discrete Regressors; Proportions
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • G01 - Financial Economics - - General - - - Financial Crises

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecb:ecbwps:20172025. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Official Publications (email available below). General contact details of provider: https://edirc.repec.org/data/emieude.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.