On policymakers' loss function and the evaluation of early warning systems
AbstractThis paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers' preferences between issuing false alarms and missing crises, as well as individual observations. The novelty derives from three enhancements: i) accounting for unconditional probabilities of the classes, ii) computing the proportion of available Usefulness that the model captures, and iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess signals issued by any type of EWS, such as logit and probit analysis and the signaling approach, and flexible for any type of crisis EWSs, such as banking, debt and currency crises. Applications to two renowned EWSs, and comparisons to two commonly used evaluation measures, illustrate three key implications of the new measures: i) further highlights the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful ii) the need to be more concerned about the rare class and iii) the importance of correctly classifying observations of the most relevant entities. Beyond financial stability surveillance, this paper also opens the door for cost-sensitive evaluations of predictive models in other tasks. JEL Classification: E44, E58, F01, F37, G01
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by European Central Bank in its series Working Paper Series with number 1509.
Date of creation: Feb 2013
Date of revision:
Contact details of provider:
Postal: Postfach 16 03 19, Frankfurt am Main, Germany
Phone: +49 69 1344 0
Fax: +49 69 1344 6000
Web page: http://www.ecb.europa.eu/home/html/index.en.html
More information through EDIRC
Postal: Press and Information Division, European Central Bank, Kaiserstrasse 29, 60311 Frankfurt am Main, Germany
Other versions of this item:
- Sarlin, Peter, 2013. "On policymakers’ loss functions and the evaluation of early warning systems," Economics Letters, Elsevier, vol. 119(1), pages 1-7.
- E44 - Macroeconomics and Monetary Economics - - Money and Interest Rates - - - Financial Markets and the Macroeconomy
- E58 - Macroeconomics and Monetary Economics - - Monetary Policy, Central Banking, and the Supply of Money and Credit - - - Central Banks and Their Policies
- F01 - International Economics - - General - - - Global Outlook
- F37 - International Economics - - International Finance - - - International Finance Forecasting and Simulation: Models and Applications
- G01 - Financial Economics - - General - - - Financial Crises
This paper has been announced in the following NEP Reports:
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Mathias Drehmann & Claudio Borio & Kostas Tsatsaronis, 2011.
"Anchoring countercyclical capital buffers: the role of credit aggregates,"
BIS Working Papers
355, Bank for International Settlements.
- Mathias Drehmann & Claudio Borio & Kostas Tsatsaronis, 2011. "Anchoring Countercyclical Capital Buffers: The role of Credit Aggregates," International Journal of Central Banking, International Journal of Central Banking, vol. 7(4), pages 189-240, December.
- Kaminsky, Graciela & Lizondo, Saul & Reinhart, Carmen M., 1997.
"Leading indicators of currency crises,"
Policy Research Working Paper Series
1852, The World Bank.
- Frankel, Jeffrey A. & Rose, Andrew K., 1996.
"Currency crashes in emerging markets: An empirical treatment,"
Journal of International Economics,
Elsevier, vol. 41(3-4), pages 351-366, November.
- Jeffrey A. Frankel & Andrew K. Rose, 1996. "Currency crashes in emerging markets: an empirical treatment," International Finance Discussion Papers 534, Board of Governors of the Federal Reserve System (U.S.).
- Abhyankar, Abhay & Sarno, Lucio & Valente, Giorgio, 2004.
"Exchange Rates and Fundamentals: Evidence on the Economic Value of Predictability,"
CEPR Discussion Papers
4365, C.E.P.R. Discussion Papers.
- Abhyankar, Abhay & Sarno, Lucio & Valente, Giorgio, 2005. "Exchange rates and fundamentals: evidence on the economic value of predictability," Journal of International Economics, Elsevier, vol. 66(2), pages 325-348, July.
- Giorgio Valente & Lucio Sarno & Abhay Abhayankar, 2004. "Exchange Rates and Fundamentals: Evidence on the Economic Value of Predictability," Working Papers wp04-01, Warwick Business School, Finance Group.
- Sarlin, Peter & Peltonen, Tuomas A., 2011.
"Mapping the state of financial stability,"
Working Paper Series
1382, European Central Bank.
- Sarlin, Peter & Peltonen, Tuomas A., 2013. "Mapping the state of financial stability," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 26(C), pages 46-76.
- Bussiere, Matthieu & Fratzscher, Marcel, 2008. "Low probability, high impact: Policy making and extreme events," Journal of Policy Modeling, Elsevier, vol. 30(1), pages 111-121.
- Alessi, Lucia & Detken, Carsten, 2011. "Quasi real time early warning indicators for costly asset price boom/bust cycles: A role for global liquidity," European Journal of Political Economy, Elsevier, vol. 27(3), pages 520-533, September.
- Enrica Detragiache & Asli DemirgÃ¼Ã§-Kunt, 1999.
"Monitoring Banking Sector Fragility - A Multivariate Logit Approach,"
IMF Working Papers
99/147, International Monetary Fund.
- Demirguc, Asli & Detragiache, Enrica, 2000. "Monitoring Banking Sector Fragility: A Multivariate Logit Approach," World Bank Economic Review, World Bank Group, vol. 14(2), pages 287-307, May.
- Axel Schimmelpfennig & Nouriel Roubini & Paolo Manasse, 2003. "Predicting Sovereign Debt Crises," IMF Working Papers 03/221, International Monetary Fund.
- Òscar Jordà & Moritz Schularick & Alan M. Taylor, 2010.
"Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons,"
NBER Working Papers
16567, National Bureau of Economic Research, Inc.
- �scar Jord� & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan, vol. 59(2), pages 340-378, June.
- Berg, Andrew & Pattillo, Catherine, 1999. "Predicting currency crises:: The indicators approach and an alternative," Journal of International Money and Finance, Elsevier, vol. 18(4), pages 561-586, August.
- Kasper Lund-Jensen, 2012. "Monitoring Systemic Risk Based on Dynamic Thresholds," IMF Working Papers 12/159, International Monetary Fund.
- Makram El-Shagi & Tobias Knedlik & Gregor von Schweinitz, 2012.
"Predicting Financial Crises: The (Statistical) Significance of the Signals Approach,"
IWH Discussion Papers
3, Halle Institute for Economic Research.
- El-Shagi, M. & Knedlik, T. & von Schweinitz, G., 2013. "Predicting financial crises: The (statistical) significance of the signals approach," Journal of International Money and Finance, Elsevier, vol. 35(C), pages 76-103.
- Òscar Jordà & Alan M. Taylor, 2011. "Performance Evaluation of Zero Net-Investment Strategies," NBER Working Papers 17150, National Bureau of Economic Research, Inc.
- Bertrand Candelon & Elena-Ivona Dumitrescu & Christophe Hurlin, 2012. "How to Evaluate an Early-Warning System: Toward a Unified Statistical Framework for Assessing Financial Crises Forecasting Methods," IMF Economic Review, Palgrave Macmillan, vol. 60(1), pages 75-113, April.
- Betz, Frank & Oprica, Silviu & Peltonen, Tuomas A. & Sarlin, Peter, 2013. "Predicting distress in European banks," Working Paper Series 1597, European Central Bank.
- Behn, Markus & Detken, Carsten & Peltonen, Tuomas A. & Schudel, Willem, 2013. "Setting countercyclical capital buffers based on early warning models: would it work?," Working Paper Series 1604, European Central Bank.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Official Publications).
If references are entirely missing, you can add them using this form.