IDEAS home Printed from https://ideas.repec.org/a/gam/jrisks/v8y2020i4p113-d434660.html
   My bibliography  Save this article

Exploiting Distributional Temporal Difference Learning to Deal with Tail Risk

Author

Listed:
  • Peter Bossaerts

    (Brain Mind and Markets Lab, The University of Melbourne, 198 Berkeley Street, Parkville, VIC 3010, Australia)

  • Shijie Huang

    (Brain Mind and Markets Lab, The University of Melbourne, 198 Berkeley Street, Parkville, VIC 3010, Australia)

  • Nitin Yadav

    (Brain Mind and Markets Lab, The University of Melbourne, 198 Berkeley Street, Parkville, VIC 3010, Australia)

Abstract

In traditional Reinforcement Learning (RL), agents learn to optimize actions in a dynamic context based on recursive estimation of expected values. We show that this form of machine learning fails when rewards (returns) are affected by tail risk, i.e., leptokurtosis. Here, we adapt a recent extension of RL, called distributional RL (disRL), and introduce estimation efficiency, while properly adjusting for differential impact of outliers on the two terms of the RL prediction error in the updating equations. We show that the resulting “efficient distributional RL” (e-disRL) learns much faster, and is robust once it settles on a policy. Our paper also provides a brief, nontechnical overview of machine learning, focusing on RL.

Suggested Citation

  • Peter Bossaerts & Shijie Huang & Nitin Yadav, 2020. "Exploiting Distributional Temporal Difference Learning to Deal with Tail Risk," Risks, MDPI, vol. 8(4), pages 1-20, October.
    • Handle: RePEc:gam:jrisks:v:8:y:2020:i:4:p:113-:d:434660
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-9091/8/4/113/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-9091/8/4/113/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Madan, Dilip B., 2017. "Efficient estimation of expected stock price returns," Finance Research Letters, Elsevier, vol. 23(C), pages 31-38.
    2. José Curto & José Pinto & Gonçalo Tavares, 2009. "Modeling stock markets’ volatility using GARCH models with Normal, Student’s t and stable Paretian distributions," Statistical Papers, Springer, vol. 50(2), pages 311-321, March.
    3. Alexander Shapiro & Jos Berge, 2002. "Statistical inference of minimum rank factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 67(1), pages 79-94, March.
    4. Stefan Mittnik & Marc Paolella & Svetlozar Rachev, 1998. "Unconditional and Conditional Distributional Models for the Nikkei Index," Asia-Pacific Financial Markets, Springer;Japanese Association of Financial Economics and Engineering, vol. 5(2), pages 99-128, May.
    5. Dima Alberg & Haim Shalit & Rami Yosef, 2008. "Estimating stock market volatility using asymmetric GARCH models," Applied Financial Economics, Taylor & Francis Journals, vol. 18(15), pages 1201-1208.
    6. Matthias Scherer & Svetlozar T. Rachev & Young Shin Kim & Frank J. Fabozzi, 2012. "Approximation of skewed and leptokurtic return distributions," Applied Financial Economics, Taylor & Francis Journals, vol. 22(16), pages 1305-1316, August.
    7. Bollerslev, Tim, 1987. "A Conditionally Heteroskedastic Time Series Model for Speculative Prices and Rates of Return," The Review of Economics and Statistics, MIT Press, vol. 69(3), pages 542-547, August.
    8. Qiang Sun & Wen-Xin Zhou & Jianqing Fan, 2020. "Adaptive Huber Regression," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(529), pages 254-265, January.
    9. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    10. Simonato, Jean-Guy, 2012. "GARCH processes with skewed and leptokurtic innovations: Revisiting the Johnson Su case," Finance Research Letters, Elsevier, vol. 9(4), pages 213-219.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. Hambuckers & C. Heuchenne, 2017. "A robust statistical approach to select adequate error distributions for financial returns," Journal of Applied Statistics, Taylor & Francis Journals, vol. 44(1), pages 137-161, January.
    2. Klar, B. & Lindner, F. & Meintanis, S.G., 2012. "Specification tests for the error distribution in GARCH models," Computational Statistics & Data Analysis, Elsevier, vol. 56(11), pages 3587-3598.
    3. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    4. Alessio Brini & Daniele Tantari, 2021. "Deep Reinforcement Trading with Predictable Returns," Papers 2104.14683, arXiv.org, revised May 2023.
    5. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    6. Bucevska Vesna, 2013. "An Empirical Evaluation of GARCH Models in Value-at-Risk Estimation: Evidence from the Macedonian Stock Exchange," Business Systems Research, Sciendo, vol. 4(1), pages 49-64, March.
    7. Markus Haas, 2004. "Mixed Normal Conditional Heteroskedasticity," Journal of Financial Econometrics, Oxford University Press, vol. 2(2), pages 211-250.
    8. Sung Ik Kim, 2022. "ARMA–GARCH model with fractional generalized hyperbolic innovations," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-25, December.
    9. Marc S. Paolella, 2016. "Stable-GARCH Models for Financial Returns: Fast Estimation and Tests for Stability," Econometrics, MDPI, vol. 4(2), pages 1-28, May.
    10. Simon Lalancette & Jean†Guy Simonato, 2017. "The Role of the Conditional Skewness and Kurtosis in VIX Index Valuation," European Financial Management, European Financial Management Association, vol. 23(2), pages 325-354, March.
    11. Dongming Zhu & John W. Galbraith, 2009. "Forecasting Expected Shortfall with a Generalized Asymmetric Student-t Distribution," CIRANO Working Papers 2009s-24, CIRANO.
    12. Xiaochun Liu, 2017. "An integrated macro‐financial risk‐based approach to the stressed capital requirement," Review of Financial Economics, John Wiley & Sons, vol. 34(1), pages 86-98, September.
    13. Zhu, Dongming & Galbraith, John W., 2010. "A generalized asymmetric Student-t distribution with application to financial econometrics," Journal of Econometrics, Elsevier, vol. 157(2), pages 297-305, August.
    14. repec:hal:journl:peer-00834423 is not listed on IDEAS
    15. Brini, Alessio & Tantari, Daniele, 2023. "Deep reinforcement trading with predictable returns," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 622(C).
    16. Mittnik, Stefan & Paolella, Marc S. & Rachev, Svetlozar T., 2000. "Diagnosing and treating the fat tails in financial returns data," Journal of Empirical Finance, Elsevier, vol. 7(3-4), pages 389-416, November.
    17. Christophe Chorro & Dominique Guegan & Florian Ielpo & Hanjarivo Lalaharison, 2014. "Testing for Leverage Effect in Financial Returns," Documents de travail du Centre d'Economie de la Sorbonne 14022, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    18. Chorro, Christophe & Guégan, Dominique & Ielpo, Florian & Lalaharison, Hanjarivo, 2018. "Testing for leverage effects in the returns of US equities," Journal of Empirical Finance, Elsevier, vol. 48(C), pages 290-306.
    19. Christophe Chorro & Dominique Guegan & Florian Ielpo & Hanjarivo Lalaharison, 2017. "Testing for Leverage Effects in the Returns of US Equities," Post-Print halshs-00973922, HAL.
    20. Liang, Chao & Li, Yan & Ma, Feng & Wei, Yu, 2021. "Global equity market volatilities forecasting: A comparison of leverage effects, jumps, and overnight information," International Review of Financial Analysis, Elsevier, vol. 75(C).
    21. Neenu Chalissery & Suhaib Anagreh & Mohamed Nishad T. & Mosab I. Tabash, 2022. "Mapping the Trend, Application and Forecasting Performance of Asymmetric GARCH Models: A Review Based on Bibliometric Analysis," JRFM, MDPI, vol. 15(9), pages 1-23, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jrisks:v:8:y:2020:i:4:p:113-:d:434660. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.