IDEAS home Printed from https://ideas.repec.org/p/ecb/ecbwps/20131509.html
   My bibliography  Save this paper

On policymakers' loss function and the evaluation of early warning systems

Author

Listed:
  • Sarlin, Peter

Abstract

This paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers' preferences between issuing false alarms and missing crises, as well as individual observations. The novelty derives from three enhancements: i) accounting for unconditional probabilities of the classes, ii) computing the proportion of available Usefulness that the model captures, and iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess signals issued by any type of EWS, such as logit and probit analysis and the signaling approach, and flexible for any type of crisis EWSs, such as banking, debt and currency crises. Applications to two renowned EWSs, and comparisons to two commonly used evaluation measures, illustrate three key implications of the new measures: i) further highlights the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful ii) the need to be more concerned about the rare class and iii) the importance of correctly classifying observations of the most relevant entities. Beyond financial stability surveillance, this paper also opens the door for cost-sensitive evaluations of predictive models in other tasks. JEL Classification: E44, E58, F01, F37, G01

Suggested Citation

  • Sarlin, Peter, 2013. "On policymakers' loss function and the evaluation of early warning systems," Working Paper Series 1509, European Central Bank.
  • Handle: RePEc:ecb:ecbwps:20131509
    as

    Download full text from publisher

    File URL: https://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp1509.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Sarlin, Peter & Peltonen, Tuomas A., 2013. "Mapping the state of financial stability," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 26(C), pages 46-76.
    2. Alessi, Lucia & Detken, Carsten, 2011. "Quasi real time early warning indicators for costly asset price boom/bust cycles: A role for global liquidity," European Journal of Political Economy, Elsevier, vol. 27(3), pages 520-533, September.
    3. Abhyankar, Abhay & Sarno, Lucio & Valente, Giorgio, 2005. "Exchange rates and fundamentals: evidence on the economic value of predictability," Journal of International Economics, Elsevier, vol. 66(2), pages 325-348, July.
    4. Òscar Jordà & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 59(2), pages 340-378, June.
    5. Fuertes, Ana-Maria & Kalotychou, Elena, 2006. "Early warning systems for sovereign debt crises: The role of heterogeneity," Computational Statistics & Data Analysis, Elsevier, vol. 51(2), pages 1420-1441, November.
    6. Graciela Kaminsky & Saul Lizondo & Carmen M. Reinhart, 1998. "Leading Indicators of Currency Crises," IMF Staff Papers, Palgrave Macmillan, vol. 45(1), pages 1-48, March.
    7. Demirguc, Asli & Detragiache, Enrica, 2000. "Monitoring Banking Sector Fragility: A Multivariate Logit Approach," World Bank Economic Review, World Bank Group, vol. 14(2), pages 287-307, May.
    8. Bussiere, Matthieu & Fratzscher, Marcel, 2008. "Low probability, high impact: Policy making and extreme events," Journal of Policy Modeling, Elsevier, vol. 30(1), pages 111-121.
    9. Axel Schimmelpfennig & Nouriel Roubini & Paolo Manasse, 2003. "Predicting Sovereign Debt Crises," IMF Working Papers 03/221, International Monetary Fund.
    10. Mathias Drehmann & Claudio Borio & Kostas Tsatsaronis, 2011. "Anchoring Countercyclical Capital Buffers: The role of Credit Aggregates," International Journal of Central Banking, International Journal of Central Banking, vol. 7(4), pages 189-240, December.
    11. Kasper Lund-Jensen, 2012. "Monitoring Systemic Risk Basedon Dynamic Thresholds," IMF Working Papers 12/159, International Monetary Fund.
    12. El-Shagi, M. & Knedlik, T. & von Schweinitz, G., 2013. "Predicting financial crises: The (statistical) significance of the signals approach," Journal of International Money and Finance, Elsevier, vol. 35(C), pages 76-103.
    13. A. Berg & C. Pattillo, 1999. "What Caused the Asian Crises: An Early Warning System Approach," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 28(3), pages 285-334, November.
    14. Bertrand Candelon & Elena-Ivona Dumitrescu & Christophe Hurlin, 2012. "How to Evaluate an Early-Warning System: Toward a Unified Statistical Framework for Assessing Financial Crises Forecasting Methods," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 60(1), pages 75-113, April.
    15. Òscar Jordà & Alan M. Taylor, 2011. "Performance Evaluation of Zero Net-Investment Strategies," NBER Working Papers 17150, National Bureau of Economic Research, Inc.
    16. Frankel, Jeffrey A. & Rose, Andrew K., 1996. "Currency crashes in emerging markets: An empirical treatment," Journal of International Economics, Elsevier, vol. 41(3-4), pages 351-366, November.
    17. Sarlin, Peter & Peltonen, Tuomas A., 2013. "Mapping the state of financial stability," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 26(C), pages 46-76.
    18. Berg, Andrew & Pattillo, Catherine, 1999. "Predicting currency crises:: The indicators approach and an alternative," Journal of International Money and Finance, Elsevier, vol. 18(4), pages 561-586, August.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    early warning systems; misclassification costs;

    JEL classification:

    • E44 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Financial Markets and the Macroeconomy
    • E58 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Central Banks and Their Policies
    • F01 - International Economics - - General - - - Global Outlook
    • F37 - International Economics - - International Finance - - - International Finance Forecasting and Simulation: Models and Applications
    • G01 - Financial Economics - - General - - - Financial Crises

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecb:ecbwps:20131509. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Official Publications). General contact details of provider: http://edirc.repec.org/data/emieude.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.