IDEAS home Printed from https://ideas.repec.org/a/eee/jomega/v133y2025ics0305048324002135.html
   My bibliography  Save this article

Learning about tail risk: Machine learning and combination with regularization in market risk management

Author

Listed:
  • Wang, Shuai
  • Wang, Qian
  • Lu, Helen
  • Zhang, Dongxue
  • Xing, Qianyi
  • Wang, Jianzhou

Abstract

High-quality risk management is the key to ensuring the safe, efficient, and stable operation of the financial system. The current Basel Accord requires financial institutions to regularly calculate and disclose Value at Risk (VaR) and Expected Shortfall (ES) measures. However, the inaccuracy and instability of traditional risk models have reduced users' confidence. Therefore, we propose two new probabilistic deep learning frameworks for estimating VaR and ES. The trained first framework can output expectiles that are more sensitive to tail risks to map VaR and ES measures. In the second framework, we propose to approximate VaR and ES measures with spline quantile function and estimate the parameters by designing various deep learning architectures. To ensure the effectiveness of the proposed architectures, we derived the training loss and constraints for them. In addition, we solve the problem that existing machine learning risk models are difficult to estimate ES. In this way, combining various individual risk models has great potential for risk management. Therefore, we propose a regularization-based combination framework that adaptively selects and shrinks individual risk models. The developed individual methods and combinations outperform existing methods in backtesting, assisting financial institutions to allocate capital more effectively according to the Basel Capital Accord.

Suggested Citation

  • Wang, Shuai & Wang, Qian & Lu, Helen & Zhang, Dongxue & Xing, Qianyi & Wang, Jianzhou, 2025. "Learning about tail risk: Machine learning and combination with regularization in market risk management," Omega, Elsevier, vol. 133(C).
  • Handle: RePEc:eee:jomega:v:133:y:2025:i:c:s0305048324002135
    DOI: 10.1016/j.omega.2024.103249
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0305048324002135
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.omega.2024.103249?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hu, Jianming & Luo, Qingxi & Tang, Jingwei & Heng, Jiani & Deng, Yuwen, 2022. "Conformalized temporal convolutional quantile regression networks for wind power interval forecasting," Energy, Elsevier, vol. 248(C).
    2. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    3. Bams, Dennis & Blanchard, Gildas & Lehnert, Thorsten, 2017. "Volatility measures and Value-at-Risk," International Journal of Forecasting, Elsevier, vol. 33(4), pages 848-863.
    4. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    5. Giacomini, Raffaella & Komunjer, Ivana, 2005. "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 416-431, October.
    6. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    7. Jooyong Shim & Yongtae Kim & Jangtaek Lee & Changha Hwang, 2012. "Estimating value at risk with semiparametric support vector quantile regression," Computational Statistics, Springer, vol. 27(4), pages 685-700, December.
    8. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    9. J. P. Brans & Ph. Vincke, 1985. "Note---A Preference Ranking Organisation Method," Management Science, INFORMS, vol. 31(6), pages 647-656, June.
    10. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    11. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    12. Leippold, Markus & Wang, Qian & Zhou, Wenyu, 2022. "Machine learning in the Chinese stock market," Journal of Financial Economics, Elsevier, vol. 145(2), pages 64-82.
    13. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    14. Anne Opschoor & Dick van Dijk & Michel van der Wel, 2017. "Combining density forecasts using focused scoring rules," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(7), pages 1298-1313, November.
    15. Ilias Chronopoulos & Aristeidis Raftapostolos & George Kapetanios, 2024. "Forecasting Value-at-Risk Using Deep Neural Network Quantile Regression," Journal of Financial Econometrics, Oxford University Press, vol. 22(3), pages 636-669.
    16. Richard H. Gerlach & Cathy W. S. Chen & Nancy Y. C. Chan, 2011. "Bayesian Time-Varying Quantile Forecasting for Value-at-Risk in Financial Markets," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(4), pages 481-492, October.
    17. Jooyoung Jeon & James W. Taylor, 2013. "Using CAViaR Models with Implied Volatility for Value‐at‐Risk Estimation," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 32(1), pages 62-74, January.
    18. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    19. Paul H. Kupiec, 1995. "Techniques for verifying the accuracy of risk measurement models," Finance and Economics Discussion Series 95-24, Board of Governors of the Federal Reserve System (U.S.).
    20. Alexander J. McNeil & Rüdiger Frey & Paul Embrechts, 2015. "Quantitative Risk Management: Concepts, Techniques and Tools Revised edition," Economics Books, Princeton University Press, edition 2, number 10496.
    21. Helmut Elsinger & Alfred Lehar & Martin Summer, 2006. "Risk Assessment for Banking Systems," Management Science, INFORMS, vol. 52(9), pages 1301-1314, September.
    22. Sebastian M. Blanc & Thomas Setzer, 2020. "Bias–Variance Trade-Off and Shrinkage of Weights in Forecast Combination," Management Science, INFORMS, vol. 66(12), pages 5720-5737, December.
    23. Thompson, Ryan & Qian, Yilin & Vasnev, Andrey L., 2024. "Flexible global forecast combinations," Omega, Elsevier, vol. 126(C).
    24. Tilmann Gneiting & Roopesh Ranjan, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 411-422, July.
    25. Gianni De Nicolò & Marcella Lucchetta, 2017. "Forecasting Tail Risks," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(1), pages 159-170, January.
    26. Meng, Xiaochun & Taylor, James W., 2018. "An approximate long-memory range-based approach for value at risk estimation," International Journal of Forecasting, Elsevier, vol. 34(3), pages 377-388.
    27. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    28. Guo, Mengzhuo & Zhang, Qingpeng & Liao, Xiuwu & Chen, Frank Youhua & Zeng, Daniel Dajun, 2021. "A hybrid machine learning framework for analyzing human decision-making through learning preferences," Omega, Elsevier, vol. 101(C).
    29. Paul Embrechts & Haiyan Liu & Tiantian Mao & Ruodu Wang, 2017. "Quantile-Based Risk Sharing with Heterogeneous Beliefs," Swiss Finance Institute Research Paper Series 17-65, Swiss Finance Institute, revised Jan 2018.
    30. Wang, Jujie & Zhuang, Zhenzhen & Gao, Dongming, 2023. "An enhanced hybrid model based on multiple influencing factors and divide-conquer strategy for carbon price prediction," Omega, Elsevier, vol. 120(C).
    31. Hiroshi Shiraishi & Tomoshige Nakamura & Ryotato Shibuki, 2024. "Time Series Quantile Regression Using Random Forests," Journal of Time Series Analysis, Wiley Blackwell, vol. 45(4), pages 639-659, July.
    32. Patton, Andrew J. & Ziegel, Johanna F. & Chen, Rui, 2019. "Dynamic semiparametric models for expected shortfall (and Value-at-Risk)," Journal of Econometrics, Elsevier, vol. 211(2), pages 388-413.
    33. McNeil, Alexander J. & Frey, Rudiger, 2000. "Estimation of tail-related risk measures for heteroscedastic financial time series: an extreme value approach," Journal of Empirical Finance, Elsevier, vol. 7(3-4), pages 271-300, November.
    34. Robert F. Engle & Simone Manganelli, 2004. "CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles," Journal of Business & Economic Statistics, American Statistical Association, vol. 22, pages 367-381, October.
    35. Georg Keilbar & Weining Wang, 2022. "Modelling systemic risk using neural network quantile regression," Empirical Economics, Springer, vol. 62(1), pages 93-118, January.
    36. Gibbs, Christopher G. & Vasnev, Andrey L., 2024. "Conditionally optimal weights and forward-looking approaches to combining forecasts," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1734-1751.
    37. Fuertes, Ana-Maria & Olmo, Jose, 2013. "Optimally harnessing inter-day and intra-day information for daily value-at-risk prediction," International Journal of Forecasting, Elsevier, vol. 29(1), pages 28-42.
    38. van der Meer, Dennis & Pinson, Pierre & Camal, Simon & Kariniotakis, Georges, 2024. "CRPS-based online learning for nonlinear probabilistic forecast combination," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1449-1466.
    39. Walter Pohl & Karl Schmedders & Ole Wilms, 2018. "Higher Order Effects in Asset Pricing Models with Long‐Run Risks," Journal of Finance, American Finance Association, vol. 73(3), pages 1061-1111, June.
    40. Halbleib, Roxana & Pohlmeier, Winfried, 2012. "Improving the value at risk forecasts: Theory and evidence from the financial crisis," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1212-1228.
    41. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    42. Bollerslev, Tim, 1987. "A Conditionally Heteroskedastic Time Series Model for Speculative Prices and Rates of Return," The Review of Economics and Statistics, MIT Press, vol. 69(3), pages 542-547, August.
    43. Gneiting, Tilmann & Ranjan, Roopesh, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(3), pages 411-422.
    44. Sehgal, Ruchika & Sharma, Amita & Mansini, Renata, 2023. "Worst-case analysis of Omega-VaR ratio optimization model," Omega, Elsevier, vol. 114(C).
    45. Gu, Shihao & Kelly, Bryan & Xiu, Dacheng, 2021. "Autoencoder asset pricing models," Journal of Econometrics, Elsevier, vol. 222(1), pages 429-450.
    46. Taylor, James W., 2020. "Forecast combinations for value at risk and expected shortfall," International Journal of Forecasting, Elsevier, vol. 36(2), pages 428-441.
    47. Pritsker, Matthew, 2006. "The hidden dangers of historical simulation," Journal of Banking & Finance, Elsevier, vol. 30(2), pages 561-582, February.
    48. Jiang, Ping & Liu, Zhenkun & Abedin, Mohammad Zoynul & Wang, Jianzhou & Yang, Wendong & Dong, Qingli, 2024. "Profit-driven weighted classifier with interpretable ability for customer churn prediction," Omega, Elsevier, vol. 125(C).
    49. Helmut Elsinger & Alfred Lehar & Martin Summer, 2006. "Using Market Information for Banking System Risk Assessment," International Journal of Central Banking, International Journal of Central Banking, vol. 2(1), March.
    50. Tobias Fissler & Johanna F. Ziegel & Tilmann Gneiting, 2015. "Expected Shortfall is jointly elicitable with Value at Risk - Implications for backtesting," Papers 1507.00244, arXiv.org, revised Jul 2015.
    51. James W. Taylor, 2019. "Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(1), pages 121-133, January.
    52. Newey, Whitney K & Powell, James L, 1987. "Asymmetric Least Squares Estimation and Testing," Econometrica, Econometric Society, vol. 55(4), pages 819-847, July.
    53. Fangda Liu & Ruodu Wang, 2021. "A Theory for Measures of Tail Risk," Mathematics of Operations Research, INFORMS, vol. 46(3), pages 1109-1128, August.
    54. James W. Taylor, 2008. "Estimating Value at Risk and Expected Shortfall Using Expectiles," Journal of Financial Econometrics, Oxford University Press, vol. 6(2), pages 231-252, Spring.
    55. Michaël Zamo & Liliane Bel & Olivier Mestre, 2021. "Sequential aggregation of probabilistic forecasts—Application to wind speed ensemble forecasts," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 70(1), pages 202-225, January.
    56. Matsypura, Dmytro & Thompson, Ryan & Vasnev, Andrey L., 2018. "Optimal selection of expert forecasts with integer programming," Omega, Elsevier, vol. 78(C), pages 165-175.
    57. James W. Taylor & Derek W. Bunn, 1999. "A Quantile Regression Approach to Generating Prediction Intervals," Management Science, INFORMS, vol. 45(2), pages 225-237, February.
    58. Yao, Qiwei & Tong, Howell, 1996. "Asymmetric least squares regression estimation: a nonparametric approach," LSE Research Online Documents on Economics 19423, London School of Economics and Political Science, LSE Library.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Taylor, James W., 2020. "Forecast combinations for value at risk and expected shortfall," International Journal of Forecasting, Elsevier, vol. 36(2), pages 428-441.
    2. Le, Trung H., 2020. "Forecasting value at risk and expected shortfall with mixed data sampling," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1362-1379.
    3. David Happersberger & Harald Lohre & Ingmar Nolte, 2020. "Estimating portfolio risk for tail risk protection strategies," European Financial Management, European Financial Management Association, vol. 26(4), pages 1107-1146, September.
    4. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    5. Timo Dimitriadis & iaochun Liu & Julie Schnaitmann, 2023. "Encompassing Tests for Value at Risk and Expected Shortfall Multistep Forecasts Based on Inference on the Boundary," Journal of Financial Econometrics, Oxford University Press, vol. 21(2), pages 412-444.
    6. Lazar, Emese & Xue, Xiaohan, 2020. "Forecasting risk measures using intraday data in a generalized autoregressive score framework," International Journal of Forecasting, Elsevier, vol. 36(3), pages 1057-1072.
    7. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    8. Ardia, David & Bluteau, Keven & Boudt, Kris & Catania, Leopoldo, 2018. "Forecasting risk with Markov-switching GARCH models:A large-scale performance study," International Journal of Forecasting, Elsevier, vol. 34(4), pages 733-747.
    9. Alessandra Amendola & Vincenzo Candila & Antonio Naimoli & Giuseppe Storti, 2024. "Adaptive combinations of tail-risk forecasts," Papers 2406.06235, arXiv.org.
    10. Arian, Hamid & Moghimi, Mehrdad & Tabatabaei, Ehsan & Zamani, Shiva, 2022. "Encoded Value-at-Risk: A machine learning approach for portfolio risk measurement," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 202(C), pages 500-525.
    11. Federico Gatta & Fabrizio Lillo & Piero Mazzarisi, 2024. "CAESar: Conditional Autoregressive Expected Shortfall," Papers 2407.06619, arXiv.org.
    12. Bayer, Sebastian, 2018. "Combining Value-at-Risk forecasts using penalized quantile regressions," Econometrics and Statistics, Elsevier, vol. 8(C), pages 56-77.
    13. d’Addona, Stefano & Khanom, Najrin, 2022. "Estimating tail-risk using semiparametric conditional variance with an application to meme stocks," International Review of Economics & Finance, Elsevier, vol. 82(C), pages 241-260.
    14. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    15. Merlo, Luca & Petrella, Lea & Raponi, Valentina, 2021. "Forecasting VaR and ES using a joint quantile regression and its implications in portfolio allocation," Journal of Banking & Finance, Elsevier, vol. 133(C).
    16. Vica Tendenan & Richard Gerlach & Chao Wang, 2020. "Tail risk forecasting using Bayesian realized EGARCH models," Papers 2008.05147, arXiv.org, revised Aug 2020.
    17. Ning Zhang & Yujing Gong & Xiaohan Xue, 2023. "Less disagreement, better forecasts: Adjusted risk measures in the energy futures market," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 43(10), pages 1332-1372, October.
    18. Xu, Qifa & Chen, Lu & Jiang, Cuixia & Yu, Keming, 2020. "Mixed data sampling expectile regression with applications to measuring financial risk," Economic Modelling, Elsevier, vol. 91(C), pages 469-486.
    19. Luca Merlo & Lea Petrella & Valentina Raponi, 2021. "Forecasting VaR and ES using a joint quantile regression and implications in portfolio allocation," Papers 2106.06518, arXiv.org.
    20. Taylor, James W., 2022. "Forecasting Value at Risk and expected shortfall using a model with a dynamic omega ratio," Journal of Banking & Finance, Elsevier, vol. 140(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jomega:v:133:y:2025:i:c:s0305048324002135. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/375/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.