IDEAS home Printed from https://ideas.repec.org/p/inu/caeprp/2012003.html
   My bibliography  Save this paper

Pitfalls in Backtesting Historical Simulation VaR Models

Author

Listed:
  • Juan Carlos Escanciano

    (Indiana University)

  • Pei Pei

    (Indiana University and Chinese Academy of Finance and Development, Central University of Finance and Economics)

Abstract

Historical Simulation (HS) and its variant, the Filtered Historical Simulation (FHS), are the most widely used Value-at-Risk forecast methods at commercial banks. These forecast methods are traditionally evaluated by means of the unconditional backtest. This paper formally shows that the unconditional backtest is always inconsistent for backtesting HS and FHS models, with a power function that can be even smaller than the nominal level in large samples. Our ndings have fundamental implications in the determination of market risk capital requirements, and also explain Monte Carlo and empirical findings in previous studies. We also propose a data-driven weighted backtest with good power properties to evaluate HS and FHS forecasts. Finally, our theoretical findings are confirmed in a Monte Carlo simulation study and an empirical application with three U.S. stocks. The empirical application shows that multiplication factors computed under the current regulatory framework are downward biased, as they inherit the inconsistency of the unconditional backtest.

Suggested Citation

  • Juan Carlos Escanciano & Pei Pei, 2012. "Pitfalls in Backtesting Historical Simulation VaR Models," CAEPR Working Papers 2012-003, Center for Applied Economics and Policy Research, Department of Economics, Indiana University Bloomington.
  • Handle: RePEc:inu:caeprp:2012003
    as

    Download full text from publisher

    File URL: https://caepr.indiana.edu/RePEc/inu/caeprp/caepr2012-003.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jeremy Berkowitz & Peter Christoffersen & Denis Pelletier, 2011. "Evaluating Value-at-Risk Models with Desk-Level Data," Management Science, INFORMS, vol. 57(12), pages 2213-2227, December.
    2. Peter Christoffersen, 2004. "Backtesting Value-at-Risk: A Duration-Based Approach," The Journal of Financial Econometrics, Society for Financial Econometrics, vol. 2(1), pages 84-108.
    3. Christoffersen, Peter F, 1998. "Evaluating Interval Forecasts," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 841-862, November.
    4. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2006. "Volatility and Correlation Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 15, pages 777-878, Elsevier.
    5. M.J.B. Hall, 1996. "The amendment to the capital accord to incorporate market risk," BNL Quarterly Review, Banca Nazionale del Lavoro, vol. 49(197), pages 271-277.
    6. Gourieroux, C. & Jasiak, J., 2008. "Dynamic quantile models," Journal of Econometrics, Elsevier, vol. 147(1), pages 198-205, November.
    7. Sun, Shuxia, 2006. "The Bahadur representation for sample quantiles under weak dependence," Statistics & Probability Letters, Elsevier, vol. 76(12), pages 1238-1244, July.
    8. Christophe Pérignon & R.D. Smith, 2008. "A New Approach to Comparing VaR Estimation Methods," Post-Print hal-00854087, HAL.
    9. Sean D. Campbell, 2005. "A review of backtesting and backtesting procedures," Finance and Economics Discussion Series 2005-21, Board of Governors of the Federal Reserve System (U.S.).
    10. Pritsker, Matthew, 2006. "The hidden dangers of historical simulation," Journal of Banking & Finance, Elsevier, vol. 30(2), pages 561-582, February.
    11. Pérignon, Christophe & Smith, Daniel R., 2010. "The level and quality of Value-at-Risk disclosure by commercial banks," Journal of Banking & Finance, Elsevier, vol. 34(2), pages 362-377, February.
    12. Hansen, Bruce E, 1994. "Autoregressive Conditional Density Estimation," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 35(3), pages 705-730, August.
    13. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    14. Giovanni Barone-Adesi & Kostas Giannopoulos, 2001. "Non parametric VaR Techniques. Myths and Realities," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 30(2), pages 167-181, July.
    15. Robert F. Engle & Simone Manganelli, 2004. "CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles," Journal of Business & Economic Statistics, American Statistical Association, vol. 22, pages 367-381, October.
    16. Escanciano, J. Carlos & Olmo, Jose, 2010. "Backtesting Parametric Value-at-Risk With Estimation Risk," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(1), pages 36-51.
    17. Carrasco, Marine & Chen, Xiaohong, 2002. "Mixing And Moment Properties Of Various Garch And Stochastic Volatility Models," Econometric Theory, Cambridge University Press, vol. 18(1), pages 17-39, February.
    18. Koenker, Roger & Xiao, Zhijie, 2006. "Quantile Autoregression," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 980-990, September.
    19. Jeremy Berkowitz & James O'Brien, 2002. "How Accurate Are Value‐at‐Risk Models at Commercial Banks?," Journal of Finance, American Finance Association, vol. 57(3), pages 1093-1111, June.
    20. Paul H. Kupiec, 1995. "Techniques for verifying the accuracy of risk measurement models," Finance and Economics Discussion Series 95-24, Board of Governors of the Federal Reserve System (U.S.).
    21. Li, W K & Ling, Shiqing & McAleer, Michael, 2002. "Recent Theoretical Results for Time Series Models with GARCH Errors," Journal of Economic Surveys, Wiley Blackwell, vol. 16(3), pages 245-269, July.
    22. W. K. Li & Shiqing Ling & Michael McAleer, 2002. "Recent Theoretical Results for Time Series Models with GARCH Errors," Journal of Economic Surveys, Wiley Blackwell, vol. 16(3), pages 245-269, July.
    23. Delgado, Miguel A. & Carlos Escanciano, J., 2007. "Nonparametric tests for conditional symmetry in dynamic models," Journal of Econometrics, Elsevier, vol. 141(2), pages 652-682, December.
    24. Yoshihara, Ken-ichi, 1995. "The Bahadur representation of sample quantiles for sequences of strongly mixing random variables," Statistics & Probability Letters, Elsevier, vol. 24(4), pages 299-304, September.
    25. Qi Li & Jeffrey Scott Racine, 2006. "Nonparametric Econometrics: Theory and Practice," Economics Books, Princeton University Press, edition 1, volume 1, number 8355.
    26. J. Carlos Escanciano & Jose Olmo, 2011. "Robust Backtesting Tests for Value-at-risk Models," The Journal of Financial Econometrics, Society for Financial Econometrics, vol. 9(1), pages 132-161, Winter.
    27. Francq, Christian & Zakoïan, Jean-Michel, 2006. "Mixing Properties Of A General Class Of Garch(1,1) Models Without Moment Assumptions On The Observed Process," Econometric Theory, Cambridge University Press, vol. 22(5), pages 815-834, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Radu Tunaru, 2015. "Model Risk in Financial Markets:From Financial Engineering to Risk Management," World Scientific Books, World Scientific Publishing Co. Pte. Ltd., number 9524, Juni.
    2. Lyu, Yongjian & Wang, Peng & Wei, Yu & Ke, Rui, 2017. "Forecasting the VaR of crude oil market: Do alternative distributions help?," Energy Economics, Elsevier, vol. 66(C), pages 523-534.
    3. Jean-Paul Laurent & Hassan Omidi Firouzi, 2022. "Market Risk and Volatility Weighted Historical Simulation After Basel III," Working Papers hal-03679434, HAL.
    4. Juan Carlos Escanciano & Zaichao Du, 2015. "Backtesting Expected Shortfall: Accounting for Tail Risk," CAEPR Working Papers 2015-001, Center for Applied Economics and Policy Research, Department of Economics, Indiana University Bloomington.
    5. Onder Buberkoku, 2018. "Examining the Value-at-risk Performance of Fractionally Integrated GARCH Models: Evidence from Energy Commodities," International Journal of Economics and Financial Issues, Econjournals, vol. 8(3), pages 36-50.
    6. Durán Santomil, Pablo & Otero González, Luís & Martorell Cunill, Onofre & Merigó Lindahl, José M., 2018. "Backtesting an equity risk model under Solvency II," Journal of Business Research, Elsevier, vol. 89(C), pages 216-222.
    7. D. Th. Vezeris & C. J. Schinas & Th. S. Kyrgos & V. A. Bizergianidou & I. P. Karkanis, 2020. "Optimization of Backtesting Techniques in Automated High Frequency Trading Systems Using the d-Backtest PS Method," Computational Economics, Springer;Society for Computational Economics, vol. 56(4), pages 975-1054, December.
    8. Yun-Tao Shi & Xiang Xiang & Li Wang & Yuan Zhang & De-Hui Sun, 2018. "Stochastic Model Predictive Fault Tolerant Control Based on Conditional Value at Risk for Wind Energy Conversion System," Energies, MDPI, vol. 11(1), pages 1-20, January.
    9. Zaichao Du & Juan Carlos Escanciano, 2017. "Backtesting Expected Shortfall: Accounting for Tail Risk," Management Science, INFORMS, vol. 63(4), pages 940-958, April.
    10. Boucher, Christophe M. & Daníelsson, Jón & Kouontchou, Patrick S. & Maillet, Bertrand B., 2014. "Risk models-at-risk," Journal of Banking & Finance, Elsevier, vol. 44(C), pages 72-92.
    11. Ziggel, Daniel & Berens, Tobias & Weiß, Gregor N.F. & Wied, Dominik, 2014. "A new set of improved Value-at-Risk backtests," Journal of Banking & Finance, Elsevier, vol. 48(C), pages 29-41.
    12. Ilhami KARAHANOGLU, 2020. "The VaR comparison of the fresh investment toolBITCOIN with other conventional investment tools, gold, stock exchange (BIST100) and foreign currencies (EUR/USD VS TRL)," Eastern Journal of European Studies, Centre for European Studies, Alexandru Ioan Cuza University, vol. 11, pages 160-181, December.
    13. Christian Brownlees & Giuseppe Cavaliere & Alice Monti, 2018. "Evaluating The Accuracy Of Tail Risk Forecasts For Systemic Risk Measurement," Annals of Financial Economics (AFE), World Scientific Publishing Co. Pte. Ltd., vol. 13(02), pages 1-25, June.
    14. Bei, Shuhua & Yang, Aijun & Pei, Haotian & Si, Xiaoli, 2023. "Price Risk Analysis using GARCH Family Models: Evidence from Shanghai Crude Oil Futures Market," Economic Modelling, Elsevier, vol. 125(C).
    15. Guillén, Montserrat & Sarabia, José María & Prieto, Faustino, 2013. "Simple risk measure calculations for sums of positive random variables," Insurance: Mathematics and Economics, Elsevier, vol. 53(1), pages 273-280.
    16. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    17. Taras Bodnar & Vilhelm Niklasson & Erik Thors'en, 2022. "Volatility Sensitive Bayesian Estimation of Portfolio VaR and CVaR," Papers 2205.01444, arXiv.org.
    18. Lazar, Emese & Zhang, Ning, 2019. "Model risk of expected shortfall," Journal of Banking & Finance, Elsevier, vol. 105(C), pages 74-93.
    19. James Ming Chen, 2018. "On Exactitude in Financial Regulation: Value-at-Risk, Expected Shortfall, and Expectiles," Risks, MDPI, vol. 6(2), pages 1-28, June.
    20. Gregor Wei{ss} & Marcus Scheffer, 2012. "Smooth Nonparametric Bernstein Vine Copulas," Papers 1210.2043, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Escanciano, Juan Carlos & Pei, Pei, 2012. "Pitfalls in backtesting Historical Simulation VaR models," Journal of Banking & Finance, Elsevier, vol. 36(8), pages 2233-2244.
    2. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    3. Benjamin R. Auer & Benjamin Mögel, 2016. "How Accurate are Modern Value-at-Risk Estimators Derived from Extreme Value Theory?," CESifo Working Paper Series 6288, CESifo.
    4. Benjamin Mögel & Benjamin R. Auer, 2018. "How accurate are modern Value-at-Risk estimators derived from extreme value theory?," Review of Quantitative Finance and Accounting, Springer, vol. 50(4), pages 979-1030, May.
    5. Boucher, Christophe M. & Daníelsson, Jón & Kouontchou, Patrick S. & Maillet, Bertrand B., 2014. "Risk models-at-risk," Journal of Banking & Finance, Elsevier, vol. 44(C), pages 72-92.
    6. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    7. Colletaz, Gilbert & Hurlin, Christophe & Pérignon, Christophe, 2013. "The Risk Map: A new tool for validating risk models," Journal of Banking & Finance, Elsevier, vol. 37(10), pages 3843-3854.
    8. Argyropoulos, Christos & Panopoulou, Ekaterini, 2019. "Backtesting VaR and ES under the magnifying glass," International Review of Financial Analysis, Elsevier, vol. 64(C), pages 22-37.
    9. Escanciano, J. Carlos & Olmo, Jose, 2010. "Backtesting Parametric Value-at-Risk With Estimation Risk," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(1), pages 36-51.
    10. Georges Tsafack & James Cataldo, 2021. "Backtesting and estimation error: value-at-risk overviolation rate," Empirical Economics, Springer, vol. 61(3), pages 1351-1396, September.
    11. Leccadito, Arturo & Boffelli, Simona & Urga, Giovanni, 2014. "Evaluating the accuracy of value-at-risk forecasts: New multilevel tests," International Journal of Forecasting, Elsevier, vol. 30(2), pages 206-216.
    12. Escanciano, J. C. & Olmo, J., 2007. "Estimation risk effects on backtesting for parametric value-at-risk models," Working Papers 07/11, Department of Economics, City University London.
    13. Jean-Paul Laurent & Hassan Omidi Firouzi, 2022. "Market Risk and Volatility Weighted Historical Simulation After Basel III," Working Papers hal-03679434, HAL.
    14. Lazar, Emese & Zhang, Ning, 2019. "Model risk of expected shortfall," Journal of Banking & Finance, Elsevier, vol. 105(C), pages 74-93.
    15. Fries, Christian P. & Nigbur, Tobias & Seeger, Norman, 2017. "Displaced relative changes in historical simulation: Application to risk measures of interest rates with phases of negative rates," Journal of Empirical Finance, Elsevier, vol. 42(C), pages 175-198.
    16. Jeremy Berkowitz & Peter Christoffersen & Denis Pelletier, 2011. "Evaluating Value-at-Risk Models with Desk-Level Data," Management Science, INFORMS, vol. 57(12), pages 2213-2227, December.
    17. James M. O'Brien & Pawel J. Szerszen, 2014. "An Evaluation of Bank VaR Measures for Market Risk During and Before the Financial Crisis," Finance and Economics Discussion Series 2014-21, Board of Governors of the Federal Reserve System (U.S.).
    18. Wagner Piazza Gaglianone & Luiz Renato Lima & Oliver Linton & Daniel R. Smith, 2011. "Evaluating Value-at-Risk Models via Quantile Regression," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(1), pages 150-160, January.
    19. O’Brien, James & Szerszeń, Paweł J., 2017. "An evaluation of bank measures for market risk before, during and after the financial crisis," Journal of Banking & Finance, Elsevier, vol. 80(C), pages 215-234.
    20. Farkas, Walter & Fringuellotti, Fulvia & Tunaru, Radu, 2020. "A cost-benefit analysis of capital requirements adjusted for model risk," Journal of Corporate Finance, Elsevier, vol. 65(C).

    More about this item

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • G21 - Financial Economics - - Financial Institutions and Services - - - Banks; Other Depository Institutions; Micro Finance Institutions; Mortgages
    • G32 - Financial Economics - - Corporate Finance and Governance - - - Financing Policy; Financial Risk and Risk Management; Capital and Ownership Structure; Value of Firms; Goodwill

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inu:caeprp:2012003. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Center for Applied Economics and Policy Research (email available below). General contact details of provider: https://edirc.repec.org/data/caeprus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.