Value-at-risk forecasts under scrutiny—the German experience

We present an analysis of the VaR forecasts and the P&L series of all 12 German banks that used internal models for regulatory purposes throughout the period from the beginning of 2001 to the end of 2004. One task of a supervisor is to estimate the 'recalibration factor', i.e. by how much a bank over- or underestimates its VaR. The Basel traffic light approach to backtesting, which maps the count of exceptions in the trailing year to a multiplicative penalty factor, can be viewed as a way to estimate the 'recalibration factor'. We introduce techniques that provide a much more powerful inference on the recalibration factor than the Basel approach based on the count of exceptions. The notions 'return on VaR (RoVaR)' and 'well-behaved forecast system' are keys to linking the problem at hand to the established literature on the evaluation of density forecasts. We perform extensive bootstrapping analyses allowing (1) an assessment of the accuracy of our estimates of the recalibration factor and (2) a comparison of the estimation error of different scale and quantile estimators. Certain robust estimators turn out to outperform the more popular estimators used in the literature. Empirical results for the non-public data are compared to the corresponding results for hypothetical portfolios based on publicly available market data. While these comparisons have to be interpreted with care since the banks' P&L data tend to be more contaminated with errors than the major market indices, they shed light on the similarities and differences between banks' RoVaRs and market index returns.