IDEAS home Printed from https://ideas.repec.org/a/eee/jomega/v133y2025ics0305048324002135.html

Learning about tail risk: Machine learning and combination with regularization in market risk management

Author

Listed:
  • Wang, Shuai
  • Wang, Qian
  • Lu, Helen
  • Zhang, Dongxue
  • Xing, Qianyi
  • Wang, Jianzhou

Abstract

High-quality risk management is the key to ensuring the safe, efficient, and stable operation of the financial system. The current Basel Accord requires financial institutions to regularly calculate and disclose Value at Risk (VaR) and Expected Shortfall (ES) measures. However, the inaccuracy and instability of traditional risk models have reduced users' confidence. Therefore, we propose two new probabilistic deep learning frameworks for estimating VaR and ES. The trained first framework can output expectiles that are more sensitive to tail risks to map VaR and ES measures. In the second framework, we propose to approximate VaR and ES measures with spline quantile function and estimate the parameters by designing various deep learning architectures. To ensure the effectiveness of the proposed architectures, we derived the training loss and constraints for them. In addition, we solve the problem that existing machine learning risk models are difficult to estimate ES. In this way, combining various individual risk models has great potential for risk management. Therefore, we propose a regularization-based combination framework that adaptively selects and shrinks individual risk models. The developed individual methods and combinations outperform existing methods in backtesting, assisting financial institutions to allocate capital more effectively according to the Basel Capital Accord.

Suggested Citation

  • Wang, Shuai & Wang, Qian & Lu, Helen & Zhang, Dongxue & Xing, Qianyi & Wang, Jianzhou, 2025. "Learning about tail risk: Machine learning and combination with regularization in market risk management," Omega, Elsevier, vol. 133(C).
  • Handle: RePEc:eee:jomega:v:133:y:2025:i:c:s0305048324002135
    DOI: 10.1016/j.omega.2024.103249
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0305048324002135
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.omega.2024.103249?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    2. Leippold, Markus & Wang, Qian & Zhou, Wenyu, 2022. "Machine learning in the Chinese stock market," Journal of Financial Economics, Elsevier, vol. 145(2), pages 64-82.
    3. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    4. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    5. Jooyoung Jeon & James W. Taylor, 2013. "Using CAViaR Models with Implied Volatility for Value‐at‐Risk Estimation," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 32(1), pages 62-74, January.
    6. Gerlach, Richard H. & Chen, Cathy W. S. & Chan, Nancy Y. C., 2011. "Bayesian Time-Varying Quantile Forecasting for Value-at-Risk in Financial Markets," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(4), pages 481-492.
    7. Meng, Xiaochun & Taylor, James W., 2018. "An approximate long-memory range-based approach for value at risk estimation," International Journal of Forecasting, Elsevier, vol. 34(3), pages 377-388.
    8. Wang, Jujie & Zhuang, Zhenzhen & Gao, Dongming, 2023. "An enhanced hybrid model based on multiple influencing factors and divide-conquer strategy for carbon price prediction," Omega, Elsevier, vol. 120(C).
    9. Hiroshi Shiraishi & Tomoshige Nakamura & Ryotato Shibuki, 2024. "Time Series Quantile Regression Using Random Forests," Journal of Time Series Analysis, Wiley Blackwell, vol. 45(4), pages 639-659, July.
    10. Patton, Andrew J. & Ziegel, Johanna F. & Chen, Rui, 2019. "Dynamic semiparametric models for expected shortfall (and Value-at-Risk)," Journal of Econometrics, Elsevier, vol. 211(2), pages 388-413.
    11. Helmut Elsinger & Alfred Lehar & Martin Summer, 2006. "Risk Assessment for Banking Systems," Management Science, INFORMS, vol. 52(9), pages 1301-1314, September.
    12. Taylor, James W., 2020. "Forecast combinations for value at risk and expected shortfall," International Journal of Forecasting, Elsevier, vol. 36(2), pages 428-441.
    13. Robert F. Engle & Simone Manganelli, 2004. "CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles," Journal of Business & Economic Statistics, American Statistical Association, vol. 22, pages 367-381, October.
    14. Jiang, Ping & Liu, Zhenkun & Abedin, Mohammad Zoynul & Wang, Jianzhou & Yang, Wendong & Dong, Qingli, 2024. "Profit-driven weighted classifier with interpretable ability for customer churn prediction," Omega, Elsevier, vol. 125(C).
    15. James W. Taylor, 2019. "Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(1), pages 121-133, January.
    16. James W. Taylor & Derek W. Bunn, 1999. "A Quantile Regression Approach to Generating Prediction Intervals," Management Science, INFORMS, vol. 45(2), pages 225-237, February.
    17. Yao, Qiwei & Tong, Howell, 1996. "Asymmetric least squares regression estimation: a nonparametric approach," LSE Research Online Documents on Economics 19423, London School of Economics and Political Science, LSE Library.
    18. Thompson, Ryan & Qian, Yilin & Vasnev, Andrey L., 2024. "Flexible global forecast combinations," Omega, Elsevier, vol. 126(C).
    19. Hu, Jianming & Luo, Qingxi & Tang, Jingwei & Heng, Jiani & Deng, Yuwen, 2022. "Conformalized temporal convolutional quantile regression networks for wind power interval forecasting," Energy, Elsevier, vol. 248(C).
    20. Bams, Dennis & Blanchard, Gildas & Lehnert, Thorsten, 2017. "Volatility measures and Value-at-Risk," International Journal of Forecasting, Elsevier, vol. 33(4), pages 848-863.
    21. Anne Opschoor & Dick van Dijk & Michel van der Wel, 2017. "Combining density forecasts using focused scoring rules," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(7), pages 1298-1313, November.
    22. Ilias Chronopoulos & Aristeidis Raftapostolos & George Kapetanios, 2024. "Forecasting Value-at-Risk Using Deep Neural Network Quantile Regression," Journal of Financial Econometrics, Oxford University Press, vol. 22(3), pages 636-669.
    23. Tilmann Gneiting & Roopesh Ranjan, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 411-422, July.
    24. Giacomini, Raffaella & Komunjer, Ivana, 2005. "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 416-431, October.
    25. Fuertes, Ana-Maria & Olmo, Jose, 2013. "Optimally harnessing inter-day and intra-day information for daily value-at-risk prediction," International Journal of Forecasting, Elsevier, vol. 29(1), pages 28-42.
    26. Halbleib, Roxana & Pohlmeier, Winfried, 2012. "Improving the value at risk forecasts: Theory and evidence from the financial crisis," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1212-1228.
    27. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    28. Walter Pohl & Karl Schmedders & Ole Wilms, 2018. "Higher Order Effects in Asset Pricing Models with Long‐Run Risks," Journal of Finance, American Finance Association, vol. 73(3), pages 1061-1111, June.
    29. Helmut Elsinger & Alfred Lehar & Martin Summer, 2006. "Using Market Information for Banking System Risk Assessment," International Journal of Central Banking, International Journal of Central Banking, vol. 2(1), March.
    30. Fangda Liu & Ruodu Wang, 2021. "A Theory for Measures of Tail Risk," Mathematics of Operations Research, INFORMS, vol. 46(3), pages 1109-1128, August.
    31. Matsypura, Dmytro & Thompson, Ryan & Vasnev, Andrey L., 2018. "Optimal selection of expert forecasts with integer programming," Omega, Elsevier, vol. 78(C), pages 165-175.
    32. Gibbs, Christopher G. & Vasnev, Andrey L., 2024. "Conditionally optimal weights and forward-looking approaches to combining forecasts," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1734-1751.
    33. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    34. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    35. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    36. Paul H. Kupiec, 1995. "Techniques for verifying the accuracy of risk measurement models," Finance and Economics Discussion Series 95-24, Board of Governors of the Federal Reserve System (U.S.).
    37. Sebastian M. Blanc & Thomas Setzer, 2020. "Bias–Variance Trade-Off and Shrinkage of Weights in Forecast Combination," Management Science, INFORMS, vol. 66(12), pages 5720-5737, December.
    38. Paul Embrechts & Haiyan Liu & Tiantian Mao & Ruodu Wang, 2017. "Quantile-Based Risk Sharing with Heterogeneous Beliefs," Swiss Finance Institute Research Paper Series 17-65, Swiss Finance Institute, revised Jan 2018.
    39. McNeil, Alexander J. & Frey, Rudiger, 2000. "Estimation of tail-related risk measures for heteroscedastic financial time series: an extreme value approach," Journal of Empirical Finance, Elsevier, vol. 7(3-4), pages 271-300, November.
    40. van der Meer, Dennis & Pinson, Pierre & Camal, Simon & Kariniotakis, Georges, 2024. "CRPS-based online learning for nonlinear probabilistic forecast combination," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1449-1466.
    41. Sehgal, Ruchika & Sharma, Amita & Mansini, Renata, 2023. "Worst-case analysis of Omega-VaR ratio optimization model," Omega, Elsevier, vol. 114(C).
    42. Gu, Shihao & Kelly, Bryan & Xiu, Dacheng, 2021. "Autoencoder asset pricing models," Journal of Econometrics, Elsevier, vol. 222(1), pages 429-450.
    43. Pritsker, Matthew, 2006. "The hidden dangers of historical simulation," Journal of Banking & Finance, Elsevier, vol. 30(2), pages 561-582, February.
    44. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    45. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    46. Jooyong Shim & Yongtae Kim & Jangtaek Lee & Changha Hwang, 2012. "Estimating value at risk with semiparametric support vector quantile regression," Computational Statistics, Springer, vol. 27(4), pages 685-700, December.
    47. J. P. Brans & Ph. Vincke, 1985. "Note---A Preference Ranking Organisation Method," Management Science, INFORMS, vol. 31(6), pages 647-656, June.
    48. Alexander J. McNeil & Rüdiger Frey & Paul Embrechts, 2015. "Quantitative Risk Management: Concepts, Techniques and Tools Revised edition," Economics Books, Princeton University Press, edition 2, number 10496, December.
    49. Gianni De Nicolò & Marcella Lucchetta, 2017. "Forecasting Tail Risks," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(1), pages 159-170, January.
    50. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    51. Guo, Mengzhuo & Zhang, Qingpeng & Liao, Xiuwu & Chen, Frank Youhua & Zeng, Daniel Dajun, 2021. "A hybrid machine learning framework for analyzing human decision-making through learning preferences," Omega, Elsevier, vol. 101(C).
    52. Georg Keilbar & Weining Wang, 2022. "Modelling systemic risk using neural network quantile regression," Empirical Economics, Springer, vol. 62(1), pages 93-118, January.
    53. Bollerslev, Tim, 1987. "A Conditionally Heteroskedastic Time Series Model for Speculative Prices and Rates of Return," The Review of Economics and Statistics, MIT Press, vol. 69(3), pages 542-547, August.
    54. Gneiting, Tilmann & Ranjan, Roopesh, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(3), pages 411-422.
    55. Tobias Fissler & Johanna F. Ziegel & Tilmann Gneiting, 2015. "Expected Shortfall is jointly elicitable with Value at Risk - Implications for backtesting," Papers 1507.00244, arXiv.org, revised Jul 2015.
    56. Newey, Whitney K & Powell, James L, 1987. "Asymmetric Least Squares Estimation and Testing," Econometrica, Econometric Society, vol. 55(4), pages 819-847, July.
    57. James W. Taylor, 2008. "Estimating Value at Risk and Expected Shortfall Using Expectiles," Journal of Financial Econometrics, Oxford University Press, vol. 6(2), pages 231-252, Spring.
    58. Michaël Zamo & Liliane Bel & Olivier Mestre, 2021. "Sequential aggregation of probabilistic forecasts—Application to wind speed ensemble forecasts," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 70(1), pages 202-225, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xing, Qianyi & Huang, Xiaojia & Wang, Kang & Wang, Jianzhou & Wang, Shuai, 2025. "MIG-EWPFS: An ensemble probabilistic wind speed forecasting system integrating multi-dimensional feature extraction, hybrid quantile regression, and Knee improved multi-objective optimization," Energy, Elsevier, vol. 324(C).
    2. Ming Che Lee, 2025. "A Hybrid EGARCH–Informer Model with Consistent Risk Calibration for Volatility and CVaR Forecasting," Mathematics, MDPI, vol. 13(19), pages 1-22, September.
    3. Garg, Divyanee & Khan, Ahmad Zaman & Mehra, Aparna, 2026. "Enhanced indexing using cumulative prospect theory utility function with expectile risk," Omega, Elsevier, vol. 139(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Taylor, James W., 2020. "Forecast combinations for value at risk and expected shortfall," International Journal of Forecasting, Elsevier, vol. 36(2), pages 428-441.
    2. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    3. Le, Trung H., 2020. "Forecasting value at risk and expected shortfall with mixed data sampling," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1362-1379.
    4. David Happersberger & Harald Lohre & Ingmar Nolte, 2020. "Estimating portfolio risk for tail risk protection strategies," European Financial Management, European Financial Management Association, vol. 26(4), pages 1107-1146, September.
    5. Timo Dimitriadis & iaochun Liu & Julie Schnaitmann, 2023. "Encompassing Tests for Value at Risk and Expected Shortfall Multistep Forecasts Based on Inference on the Boundary," Journal of Financial Econometrics, Oxford University Press, vol. 21(2), pages 412-444.
    6. Lazar, Emese & Xue, Xiaohan, 2020. "Forecasting risk measures using intraday data in a generalized autoregressive score framework," International Journal of Forecasting, Elsevier, vol. 36(3), pages 1057-1072.
    7. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    8. Alessandra Amendola & Vincenzo Candila & Antonio Naimoli & Giuseppe Storti, 2024. "Combining Value-at-Risk and Expected Shortfall forecasts via the Model Confidence Set," Papers 2406.06235, arXiv.org, revised Feb 2026.
    9. Qiu, Zhiguo & Lazar, Emese & Nakata, Keiichi, 2024. "VaR and ES forecasting via recurrent neural network-based stateful models," International Review of Financial Analysis, Elsevier, vol. 92(C).
    10. Laura Garcia‐Jorcano & Lidia Sanchis‐Marco, 2025. "Measuring the Impact of Transition Risk on Financial Markets: A Joint VaR‐ES Approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 44(6), pages 1907-1945, September.
    11. Federico Gatta & Fabrizio Lillo & Piero Mazzarisi, 2024. "CAESar: Conditional Autoregressive Expected Shortfall," Papers 2407.06619, arXiv.org, revised Jun 2026.
    12. Bayer, Sebastian, 2018. "Combining Value-at-Risk forecasts using penalized quantile regressions," Econometrics and Statistics, Elsevier, vol. 8(C), pages 56-77.
    13. Merlo, Luca & Petrella, Lea & Raponi, Valentina, 2021. "Forecasting VaR and ES using a joint quantile regression and its implications in portfolio allocation," Journal of Banking & Finance, Elsevier, vol. 133(C).
    14. Xu, Qifa & Chen, Lu & Jiang, Cuixia & Yu, Keming, 2020. "Mixed data sampling expectile regression with applications to measuring financial risk," Economic Modelling, Elsevier, vol. 91(C), pages 469-486.
    15. Candia, Claudio & Herrera, Rodrigo, 2024. "An empirical review of dynamic extreme value models for forecasting value at risk, expected shortfall and expectile," Journal of Empirical Finance, Elsevier, vol. 77(C).
    16. Ardia, David & Bluteau, Keven & Boudt, Kris & Catania, Leopoldo, 2018. "Forecasting risk with Markov-switching GARCH models:A large-scale performance study," International Journal of Forecasting, Elsevier, vol. 34(4), pages 733-747.
    17. Arian, Hamid & Moghimi, Mehrdad & Tabatabaei, Ehsan & Zamani, Shiva, 2022. "Encoded Value-at-Risk: A machine learning approach for portfolio risk measurement," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 202(C), pages 500-525.
    18. d’Addona, Stefano & Khanom, Najrin, 2022. "Estimating tail-risk using semiparametric conditional variance with an application to meme stocks," International Review of Economics & Finance, Elsevier, vol. 82(C), pages 241-260.
    19. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    20. Vica Tendenan & Richard Gerlach & Chao Wang, 2020. "Tail risk forecasting using Bayesian realized EGARCH models," Papers 2008.05147, arXiv.org, revised Aug 2020.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jomega:v:133:y:2025:i:c:s0305048324002135. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/375/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.