IDEAS home Printed from https://ideas.repec.org/a/eee/empfin/v73y2023icp251-271.html
   My bibliography  Save this article

Forecasting realized volatility with machine learning: Panel data perspective

Author

Listed:
  • Zhu, Haibin
  • Bai, Lu
  • He, Lidan
  • Liu, Zhi

Abstract

Machine learning approaches have become very popular in many fields in this big data age. This paper considers the problem of forecasting realized volatility with machine learning using high-frequency data. Instead of treating the realized volatility as a univariate time series studied by many existing works in literature, we employ panel data analysis to improve forecasting accuracy in the short term. We use six effective machine-learning methods for the realized volatility panel data. We compare our results with the traditional linear-type models under the same panel data framework and with the single time series forecasting via the same machine learning methods. The results show that the panel-data-based machine learning method (PDML) outperforms the other methods.

Suggested Citation

  • Zhu, Haibin & Bai, Lu & He, Lidan & Liu, Zhi, 2023. "Forecasting realized volatility with machine learning: Panel data perspective," Journal of Empirical Finance, Elsevier, vol. 73(C), pages 251-271.
  • Handle: RePEc:eee:empfin:v:73:y:2023:i:c:p:251-271
    DOI: 10.1016/j.jempfin.2023.07.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0927539823000683
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jempfin.2023.07.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Inoue, Atsushi & Jin, Lu & Rossi, Barbara, 2017. "Rolling window selection for out-of-sample forecasting with time-varying parameters," Journal of Econometrics, Elsevier, vol. 196(1), pages 55-67.
    3. James Ming Chen, 2021. "An Introduction to Machine Learning for Panel Data," International Advances in Economic Research, Springer;International Atlantic Economic Society, vol. 27(1), pages 1-16, February.
    4. Donaldson, R. Glen & Kamstra, Mark, 1997. "An artificial neural network-GARCH model for international stock return volatility," Journal of Empirical Finance, Elsevier, vol. 4(1), pages 17-46, January.
    5. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    6. Chang, Chia-Lin & McAleer, Michael & Tansuchat, Roengchai, 2013. "Conditional correlations and volatility spillovers between crude oil and stock index returns," The North American Journal of Economics and Finance, Elsevier, vol. 25(C), pages 116-138.
    7. Francesco Audrino & Simon D. Knaus, 2016. "Lassoing the HAR Model: A Model Selection Perspective on Realized Volatility Dynamics," Econometric Reviews, Taylor & Francis Journals, vol. 35(8-10), pages 1485-1521, December.
    8. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    9. Campbell, John Y. & Yogo, Motohiro, 2006. "Efficient tests of stock return predictability," Journal of Financial Economics, Elsevier, vol. 81(1), pages 27-60, July.
    10. Fulvio Corsi, 2009. "A Simple Approximate Long-Memory Model of Realized Volatility," Journal of Financial Econometrics, Oxford University Press, vol. 7(2), pages 174-196, Spring.
    11. Eugene F Fama & Kenneth R French & Andrew KarolyiEditor, 2020. "Comparing Cross-Section and Time-Series Factor Models," Review of Finance, European Finance Association, vol. 33(5), pages 1891-1926.
    12. Neil Shephard & Silja Kinnebrock & Ole E. Barndorff-Neilsen, 2008. "Measuring downside risk - realised semivariance," Economics Series Working Papers 382, University of Oxford, Department of Economics.
    13. Charlotte Christiansen & Maik Schmeling & Andreas Schrimpf, 2012. "A comprehensive look at financial volatility prediction by economic variables," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(6), pages 956-977, September.
    14. Kolokolov, Aleksey & Livieri, Giulia & Pirino, Davide, 2020. "Statistical inferences for price staleness," Journal of Econometrics, Elsevier, vol. 218(1), pages 32-81.
    15. Lesmond, David A & Ogden, Joseph P & Trzcinka, Charles A, 1999. "A New Estimate of Transaction Costs," The Review of Financial Studies, Society for Financial Studies, vol. 12(5), pages 1113-1141.
    16. Kim Christensen & Mathias Siggaard & Bezirgen Veliyev, 2021. "A machine learning approach to volatility forecasting," CREATES Research Papers 2021-03, Department of Economics and Business Economics, Aarhus University.
    17. McAleer, Michael & Medeiros, Marcelo C., 2008. "A multiple regime smooth transition Heterogeneous Autoregressive model for long memory and asymmetries," Journal of Econometrics, Elsevier, vol. 147(1), pages 104-119, November.
    18. Bu, Ruijun & Hizmeri, Rodrigo & Izzeldin, Marwan & Murphy, Anthony & Tsionas, Mike, 2023. "The contribution of jump signs and activity to forecasting stock price volatility," Journal of Empirical Finance, Elsevier, vol. 70(C), pages 144-164.
    19. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    20. Wang, Hui, 2019. "VIX and volatility forecasting: A new insight," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 533(C).
    21. Torben G. Andersen & Tim Bollerslev & Francis X. Diebold & Paul Labys, 2003. "Modeling and Forecasting Realized Volatility," Econometrica, Econometric Society, vol. 71(2), pages 579-625, March.
    22. Pettenuzzo, Davide & Timmermann, Allan & Valkanov, Rossen, 2014. "Forecasting stock returns under economic constraints," Journal of Financial Economics, Elsevier, vol. 114(3), pages 517-553.
    23. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    24. Mittnik, Stefan & Robinzonov, Nikolay & Spindler, Martin, 2015. "Stock market volatility: Identifying major drivers and the nature of their impact," Journal of Banking & Finance, Elsevier, vol. 58(C), pages 1-14.
    25. Amihud, Yakov, 2002. "Illiquidity and stock returns: cross-section and time-series effects," Journal of Financial Markets, Elsevier, vol. 5(1), pages 31-56, January.
    26. Friedman, Jerome H., 2002. "Stochastic gradient boosting," Computational Statistics & Data Analysis, Elsevier, vol. 38(4), pages 367-378, February.
    27. Patton, Andrew J., 2011. "Volatility forecast comparison using imperfect volatility proxies," Journal of Econometrics, Elsevier, vol. 160(1), pages 246-256, January.
    28. Michiel de Pooter & Martin Martens & Dick van Dijk, 2008. "Predicting the Daily Covariance Matrix for S&P 100 Stocks Using Intraday Data—But Which Frequency to Use?," Econometric Reviews, Taylor & Francis Journals, vol. 27(1-3), pages 199-229.
    29. Hardouvelis, Gikas A., 1987. "Macroeconomic information and stock prices," Journal of Economics and Business, Elsevier, vol. 39(2), pages 131-140, May.
    30. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    31. Fama, Eugene F & MacBeth, James D, 1973. "Risk, Return, and Equilibrium: Empirical Tests," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 607-636, May-June.
    32. Wenbo Wu & Jiaqi Chen & Zhibin (Ben) Yang & Michael L. Tindall, 2021. "A Cross-Sectional Machine Learning Approach for Hedge Fund Return Prediction and Selection," Management Science, INFORMS, vol. 67(7), pages 4577-4601, July.
    33. Farshid Abdi & Angelo Ranaldo, 2017. "A Simple Estimation of Bid-Ask Spreads from Daily Close, High, and Low Prices," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4437-4480.
    34. Ole E. Barndorff-Nielsen & Silja Kinnebrock & Neil Shephard, 2008. "Measuring downside risk — realised semivariance," CREATES Research Papers 2008-42, Department of Economics and Business Economics, Aarhus University.
    35. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    36. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    37. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    38. Andrew J. Patton & Kevin Sheppard, 2015. "Good Volatility, Bad Volatility: Signed Jumps and The Persistence of Volatility," The Review of Economics and Statistics, MIT Press, vol. 97(3), pages 683-697, July.
    39. Ole E. Barndorff-Nielsen & Neil Shephard, 2004. "Econometric Analysis of Realized Covariation: High Frequency Based Covariance, Regression, and Correlation in Financial Economics," Econometrica, Econometric Society, vol. 72(3), pages 885-925, May.
    40. Audrino, Francesco & Sigrist, Fabio & Ballinari, Daniele, 2020. "The impact of sentiment and attention measures on stock market volatility," International Journal of Forecasting, Elsevier, vol. 36(2), pages 334-357.
    41. John H. Cochrane, 2011. "Presidential Address: Discount Rates," Journal of Finance, American Finance Association, vol. 66(4), pages 1047-1108, August.
    42. Badi H. Baltagi, 2008. "Forecasting with panel data," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 27(2), pages 153-173.
    43. Michael P. Clements & Hans-Martin Krolzig, 1998. "A comparison of the forecast performance of Markov-switching and threshold autoregressive models of US GNP," Econometrics Journal, Royal Economic Society, vol. 1(Conferenc), pages 47-75.
    44. Eugene F Fama & Kenneth R French, 2020. "Comparing Cross-Section and Time-Series Factor Models," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 1891-1926.
    45. Jeremiah Green & John R. M. Hand & X. Frank Zhang, 2017. "The Characteristics that Provide Independent Information about Average U.S. Monthly Stock Returns," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4389-4436.
    46. Augustine Addo & Fidelis Sunzuoye, 2013. "The Impact of Treasury Bill Rate and Interest Rate On The Stock Market Returns: Case Of Ghana Stock Exchange," European Journal of Business and Economics, Central Bohemia University, vol. 8(2), pages 3781:8-3781, July.
    47. Yudong Wang & Zhiyuan Pan & Chongfeng Wu, 2017. "Time‐Varying Parameter Realized Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 36(5), pages 566-580, August.
    48. Chuong Luong & Nikolai Dokuchaev, 2018. "Forecasting of Realised Volatility with the Random Forests Algorithm," JRFM, MDPI, vol. 11(4), pages 1-15, October.
    49. Hu, Michael Y. & Tsoukalas, Christos, 1999. "Combining conditional volatility forecasts using neural networks: an application to the EMS exchange rates," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 9(4), pages 407-422, November.
    50. Mauro Bernardi & Leopoldo Catania, 2018. "The model confidence set package for R," International Journal of Computational Economics and Econometrics, Inderscience Enterprises Ltd, vol. 8(2), pages 144-158.
    51. Cederburg, Scott & O’Doherty, Michael S. & Wang, Feifei & Yan, Xuemin (Sterling), 2020. "On the performance of volatility-managed portfolios," Journal of Financial Economics, Elsevier, vol. 138(1), pages 95-117.
    52. Michael W. Brandt & Alon Brav & John R. Graham & Alok Kumar, 2010. "The Idiosyncratic Volatility Puzzle: Time Trend or Speculative Episodes?," The Review of Financial Studies, Society for Financial Studies, vol. 23(2), pages 863-899, February.
    53. Wang, Yudong & Wei, Yu & Wu, Chongfeng & Yin, Libo, 2018. "Oil and the short-term predictability of stock return volatility," Journal of Empirical Finance, Elsevier, vol. 47(C), pages 90-104.
    54. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    55. Yufeng Han & David Lesmond, 2011. "Liquidity Biases and the Pricing of Cross-sectional Idiosyncratic Volatility," The Review of Financial Studies, Society for Financial Studies, vol. 24(5), pages 1590-1629.
    56. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    57. Barroso, Pedro & Boons, Martijn & Karehnke, Paul, 2021. "Time-varying state variable risk premia in the ICAPM," Journal of Financial Economics, Elsevier, vol. 139(2), pages 428-451.
    58. Lyócsa, Štefan & Molnár, Peter & Výrost, Tomáš, 2021. "Stock market volatility forecasting: Do we need high-frequency data?," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1092-1110.
    59. Giuseppe Buccheri & Davide Pirino & Luca Trapin, 2021. "Managing liquidity with portfolio staleness," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 44(1), pages 215-239, June.
    60. Donaldson, R Glen & Kamstra, Mark, 1996. "A New Dividend Forecasting Procedure That Rejects Bubbles in Asset Prices: The Case of 1929's Stock Crash," The Review of Financial Studies, Society for Financial Studies, vol. 9(2), pages 333-383.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alessio Brini & Giacomo Toscano, 2024. "SpotV2Net: Multivariate Intraday Spot Volatility Forecasting via Vol-of-Vol-Informed Graph Attention Networks," Papers 2401.06249, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kim Christensen & Mathias Siggaard & Bezirgen Veliyev, 2021. "A machine learning approach to volatility forecasting," CREATES Research Papers 2021-03, Department of Economics and Business Economics, Aarhus University.
    2. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Zhang, Yaojie & Wei, Yu & Zhang, Yi & Jin, Daxiang, 2019. "Forecasting oil price volatility: Forecast combination versus shrinkage method," Energy Economics, Elsevier, vol. 80(C), pages 423-433.
    5. Chao Zhang & Yihuang Zhang & Mihai Cucuringu & Zhongmin Qian, 2022. "Volatility forecasting with machine learning and intraday commonality," Papers 2202.08962, arXiv.org, revised Feb 2023.
    6. Liu, Guangqiang & Guo, Xiaozhu, 2022. "Forecasting stock market volatility using commodity futures volatility information," Resources Policy, Elsevier, vol. 75(C).
    7. Xue Gong & Weiguo Zhang & Yuan Zhao & Xin Ye, 2023. "Forecasting stock volatility with a large set of predictors: A new forecast combination method," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1622-1647, November.
    8. Zhang, Yaojie & Lei, Likun & Wei, Yu, 2020. "Forecasting the Chinese stock market volatility with international market volatilities: The role of regime switching," The North American Journal of Economics and Finance, Elsevier, vol. 52(C).
    9. Chun, Dohyun & Cho, Hoon & Ryu, Doojin, 2023. "Discovering the drivers of stock market volatility in a data-rich world," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 82(C).
    10. Papantonis, Ioannis & Rompolis, Leonidas & Tzavalis, Elias, 2023. "Improving variance forecasts: The role of Realized Variance features," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1221-1237.
    11. Ricardo P. Masini & Marcelo C. Medeiros & Eduardo F. Mendes, 2023. "Machine learning advances for time series forecasting," Journal of Economic Surveys, Wiley Blackwell, vol. 37(1), pages 76-111, February.
    12. Niu, Zibo & Wang, Chenlu & Zhang, Hongwei, 2023. "Forecasting stock market volatility with various geopolitical risks categories: New evidence from machine learning models," International Review of Financial Analysis, Elsevier, vol. 89(C).
    13. Gong, Xue & Ye, Xin & Zhang, Weiguo & Zhang, Yue, 2023. "Predicting energy futures high-frequency volatility using technical indicators: The role of interaction," Energy Economics, Elsevier, vol. 119(C).
    14. Huei-Wen Teng & Yu-Hsien Li, 2023. "Can deep neural networks outperform Fama-MacBeth regression and other supervised learning approaches in stock returns prediction with asset-pricing factors?," Digital Finance, Springer, vol. 5(1), pages 149-182, March.
    15. He, Mengxi & Wang, Yudong & Zeng, Qing & Zhang, Yaojie, 2023. "Forecasting aggregate stock market volatility with industry volatilities: The role of spillover index," Research in International Business and Finance, Elsevier, vol. 65(C).
    16. Audrino, Francesco & Sigrist, Fabio & Ballinari, Daniele, 2020. "The impact of sentiment and attention measures on stock market volatility," International Journal of Forecasting, Elsevier, vol. 36(2), pages 334-357.
    17. Danyan Wen & Mengxi He & Yaojie Zhang & Yudong Wang, 2022. "Forecasting realized volatility of Chinese stock market: A simple but efficient truncated approach," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(2), pages 230-251, March.
    18. Wei Zhang & Kai Yan & Dehua Shen, 2021. "Can the Baidu Index predict realized volatility in the Chinese stock market?," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 7(1), pages 1-31, December.
    19. Zhang, Yaojie & Wahab, M.I.M. & Wang, Yudong, 2023. "Forecasting crude oil market volatility using variable selection and common factor," International Journal of Forecasting, Elsevier, vol. 39(1), pages 486-502.
    20. Yaojie Zhang & Yudong Wang & Feng Ma & Yu Wei, 2022. "To jump or not to jump: momentum of jumps in crude oil price volatility prediction," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-31, December.

    More about this item

    Keywords

    Realized volatility; Panel data analysis; Machine learning; Forecasting;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • G10 - Financial Economics - - General Financial Markets - - - General (includes Measurement and Data)
    • G12 - Financial Economics - - General Financial Markets - - - Asset Pricing; Trading Volume; Bond Interest Rates

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:empfin:v:73:y:2023:i:c:p:251-271. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jempfin .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.