IDEAS home Printed from https://ideas.repec.org/a/jof/jforec/v26y2007i3p203-225.html
   My bibliography  Save this article

Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003; see Bao et al. (2004)

Author

Listed:
  • Tae-Hwy Lee

    (University of California, Riverside, California USA)

  • Yong Bao

    (University of Texas, San Antonio, Texas, USA)

  • Burak Saltoğlu

    (Bosphorous University, Istanbul, Turkey)

Abstract

In this paper we discuss how to compare various (possibly misspecified) density forecast models using the Kullback-Leibler information criterion (KLIC) of a candidate density forecast model with respect to the true density. The KLIC differential between a pair of competing models is the (predictive) log-likelihood ratio (LR) between the two models. Even though the true density is unknown, using the LR statistic amounts to comparing models with the KLIC as a loss function and thus enables us to assess which density forecast model can approximate the true density more closely. We also discuss how this KLIC is related to the KLIC based on the probability integral transform (PIT) in the framework of Diebold et al. (1998). While they are asymptotically equivalent, the PIT-based KLIC is best suited for evaluating the adequacy of each density forecast model and the original KLIC is best suited for comparing competing models. In an empirical study with the S&P500 and NASDAQ daily return series, we find strong evidence for rejecting the normal-GARCH benchmark model, in favor of the models that can capture skewness in the conditional distribution and asymmetry and long memory in the conditional variance. Copyright © 2007 John Wiley & Sons, Ltd.

Suggested Citation

  • Tae-Hwy Lee & Yong Bao & Burak Saltoğlu, 2007. "Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003;," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 203-225.
  • Handle: RePEc:jof:jforec:v:26:y:2007:i:3:p:203-225
    DOI: 10.1002/for.1023
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/for.1023
    File Function: Link to full text; subscription required
    Download Restriction: no

    References listed on IDEAS

    as
    1. Jondeau, Eric & Rockinger, Michael, 2001. "Gram-Charlier densities," Journal of Economic Dynamics and Control, Elsevier, vol. 25(10), pages 1457-1483, October.
    2. Giorgio Valente & Lucio Sarno, 2004. "Comparing the accuracy of density forecasts from competing models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(8), pages 541-557.
    3. Nelson, Daniel B, 1991. "Conditional Heteroskedasticity in Asset Returns: A New Approach," Econometrica, Econometric Society, vol. 59(2), pages 347-370, March.
    4. Panayiotis Theodossiou, 1998. "Financial Data and the Skewed Generalized T Distribution," Management Science, INFORMS, vol. 44(12-Part-1), pages 1650-1661, December.
    5. Christian Gourieroux & Joann Jasiak, 2010. "Local Likelihood Density Estimation and Value-at-Risk," Journal of Probability and Statistics, Hindawi, vol. 2010, pages 1-26, June.
    6. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    7. Ser-Huang Poon & Clive W.J. Granger, 2003. "Forecasting Volatility in Financial Markets: A Review," Journal of Economic Literature, American Economic Association, vol. 41(2), pages 478-539, June.
    8. Ding, Zhuanxin & Granger, Clive W. J. & Engle, Robert F., 1993. "A long memory property of stock market returns and a new model," Journal of Empirical Finance, Elsevier, vol. 1(1), pages 83-106, June.
    9. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    10. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    11. Pascual, Lorenzo & Romo, Juan & Ruiz, Esther, 2001. "Effects of parameter estimation on prediction densities: a bootstrap approach," International Journal of Forecasting, Elsevier, vol. 17(1), pages 83-103.
    12. Bollerslev, Tim & Ole Mikkelsen, Hans, 1996. "Modeling and pricing long memory in stock market volatility," Journal of Econometrics, Elsevier, vol. 73(1), pages 151-184, July.
    13. Ding, Zhuanxin & Granger, Clive W. J., 1996. "Modeling volatility persistence of speculative returns: A new approach," Journal of Econometrics, Elsevier, vol. 73(1), pages 185-215, July.
    14. Bollerslev, Tim, 1986. "Generalized autoregressive conditional heteroskedasticity," Journal of Econometrics, Elsevier, vol. 31(3), pages 307-327, April.
    15. Torben G. Andersen & Luca Benzoni & Jesper Lund, 2002. "An Empirical Investigation of Continuous-Time Equity Return Models," Journal of Finance, American Finance Association, vol. 57(3), pages 1239-1284, June.
    16. Bauwens, Luc & Giot, Pierre & Grammig, Joachim & Veredas, David, 2004. "A comparison of financial duration models via density forecasts," International Journal of Forecasting, Elsevier, vol. 20(4), pages 589-609.
    17. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    18. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    19. Kandel, Shmuel & Stambaugh, Robert F, 1996. "On the Predictability of Stock Returns: An Asset-Allocation Perspective," Journal of Finance, American Finance Association, vol. 51(2), pages 385-424, June.
    20. Glosten, Lawrence R & Jagannathan, Ravi & Runkle, David E, 1993. "On the Relation between the Expected Value and the Volatility of the Nominal Excess Return on Stocks," Journal of Finance, American Finance Association, vol. 48(5), pages 1779-1801, December.
    21. Davidson, James, 2004. "Moment and Memory Properties of Linear Conditional Heteroscedasticity Models, and a New Model," Journal of Business & Economic Statistics, American Statistical Association, vol. 22(1), pages 16-29, January.
    22. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    23. Yihong Xia, 2001. "Learning about Predictability: The Effects of Parameter Uncertainty on Dynamic Asset Allocation," Journal of Finance, American Finance Association, vol. 56(1), pages 205-246, February.
    24. Hong, Yongmiao & Lee, Tae-Hwy, 2003. "Diagnostic Checking For The Adequacy Of Nonlinear Time Series Models," Econometric Theory, Cambridge University Press, vol. 19(6), pages 1065-1121, December.
    25. Gallant, A Ronald & Nychka, Douglas W, 1987. "Semi-nonparametric Maximum Likelihood Estimation," Econometrica, Econometric Society, vol. 55(2), pages 363-390, March.
    26. McDonald, James B. & Xu, Yexiao J., 1995. "A generalization of the beta distribution with applications," Journal of Econometrics, Elsevier, vol. 66(1-2), pages 133-152.
    27. West, Kenneth D & McCracken, Michael W, 1998. "Regression-Based Tests of Predictive Ability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 817-840, November.
    28. J. L. Knight & S. E. Satchell & K. C. Tran, 1995. "Statistical modelling of asymmetric risk in asset returns," Applied Mathematical Finance, Taylor & Francis Journals, vol. 2(3), pages 155-172.
    29. Engle, Robert F & Gonzalez-Rivera, Gloria, 1991. "Semiparametric ARCH Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 9(4), pages 345-359, October.
    30. Jackwerth, Jens Carsten & Rubinstein, Mark, 1996. "Recovering Probability Distributions from Option Prices," Journal of Finance, American Finance Association, vol. 51(5), pages 1611-1632, December.
    31. Dr. James Mitchell, 2005. "Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ÔfanÕ charts of inflation," National Institute of Economic and Social Research (NIESR) Discussion Papers 253, National Institute of Economic and Social Research.
    32. McDonald, James B. & Newey, Whitney K., 1988. "Partially Adaptive Estimation of Regression Models via the Generalized T Distribution," Econometric Theory, Cambridge University Press, vol. 4(3), pages 428-457, December.
    33. James Mitchell & Stephen G. Hall, 2005. "Evaluating, Comparing and Combining Density Forecasts Using the KLIC with an Application to the Bank of England and NIESR ‘Fan’ Charts of Inflation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 67(s1), pages 995-1033, December.
    34. Corradi, Valentina & Swanson, Norman R., 2006. "Predictive density and conditional confidence interval accuracy tests," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 187-228.
    35. Nicholas Barberis, 2000. "Investing for the Long Run when Returns Are Predictable," Journal of Finance, American Finance Association, vol. 55(1), pages 225-264, February.
    36. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    37. Subu Venkataraman, 1997. "Value at risk for a mixture of normal distributions: the use of quasi- Bayesian estimation techniques," Economic Perspectives, Federal Reserve Bank of Chicago, vol. 21(Mar), pages 2-13.
    38. Zakoian, Jean-Michel, 1994. "Threshold heteroskedastic models," Journal of Economic Dynamics and Control, Elsevier, vol. 18(5), pages 931-955, September.
    39. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    40. Sawa, Takamitsu, 1978. "Information Criteria for Discriminating among Alternative Regression Models," Econometrica, Econometric Society, vol. 46(6), pages 1273-1291, November.
    41. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    2. Lee, Tae-Hwy & Long, Xiangdong, 2009. "Copula-based multivariate GARCH model with uncorrelated dependent errors," Journal of Econometrics, Elsevier, vol. 150(2), pages 207-218, June.
    3. Maheu, John M. & McCurdy, Thomas H., 2011. "Do high-frequency measures of volatility improve forecasts of return distributions?," Journal of Econometrics, Elsevier, vol. 160(1), pages 69-76, January.
    4. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2010. "Out-of-sample comparison of copula specifications in multivariate density forecasts," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1596-1609, September.
    5. Francesco Ravazzolo & Shaun P Vahey, 2010. "Measuring Core Inflation in Australia with Disaggregate Ensembles," RBA Annual Conference Volume (Discontinued), in: Renée Fry & Callum Jones & Christopher Kent (ed.),Inflation in an Era of Relative Price Shocks, Reserve Bank of Australia.
    6. Kyungchul Song, 2009. "Testing Predictive Ability and Power Robustification," PIER Working Paper Archive 09-035, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    7. Garratt, Anthony & Mitchell, James & Vahey, Shaun P., 2014. "Measuring output gap nowcast uncertainty," International Journal of Forecasting, Elsevier, vol. 30(2), pages 268-279.
    8. Li, Yushu & Andersson, Jonas, 2014. "A Likelihood Ratio and Markov Chain Based Method to Evaluate Density Forecasting," Discussion Papers 2014/12, Norwegian School of Economics, Department of Business and Management Science.
    9. Anthony Garratt & James Mitchell & Shaun P. Vahey, 2009. "Measuring Output Gap Uncertainty," Birkbeck Working Papers in Economics and Finance 0909, Birkbeck, Department of Economics, Mathematics & Statistics.
    10. Cheng, Xixin & Li, W.K. & Yu, Philip L.H. & Zhou, Xuan & Wang, Chao & Lo, P.H., 2011. "Modeling threshold conditional heteroscedasticity with regime-dependent skewness and kurtosis," Computational Statistics & Data Analysis, Elsevier, vol. 55(9), pages 2590-2604, September.
    11. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    12. Ravazzolo Francesco & Vahey Shaun P., 2014. "Forecast densities for economic aggregates from disaggregate ensembles," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 18(4), pages 1-15, September.
    13. Del Brio, Esther B. & Ñíguez, Trino-Manuel & Perote, Javier, 2011. "Multivariate semi-nonparametric distributions with dynamic conditional correlations," International Journal of Forecasting, Elsevier, vol. 27(2), pages 347-364.
    14. Rompolis, Leonidas S., 2010. "Retrieving risk neutral densities from European option prices based on the principle of maximum entropy," Journal of Empirical Finance, Elsevier, vol. 17(5), pages 918-937, December.
    15. Hua, Jian & Manzan, Sebastiano, 2013. "Forecasting the return distribution using high-frequency volatility measures," Journal of Banking & Finance, Elsevier, vol. 37(11), pages 4381-4403.
    16. Gloria González-Rivera & Tae-Hwy Lee, 2007. "Nonlinear Time Series in Financial Forecasting," Working Papers 200803, University of California at Riverside, Department of Economics, revised Feb 2008.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2006. "Volatility and Correlation Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 15, pages 777-878, Elsevier.
    2. repec:zbw:cfswop:wp200508 is not listed on IDEAS
    3. Torben G. Andersen & Tim Bollerslev & Peter F. Christoffersen & Francis X. Diebold, 2005. "Volatility Forecasting," PIER Working Paper Archive 05-011, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    4. Halkos, George E. & Tsirivis, Apostolos S., 2019. "Effective energy commodity risk management: Econometric modeling of price volatility," Economic Analysis and Policy, Elsevier, vol. 63(C), pages 234-250.
    5. Conrad, Christian & Karanasos, Menelaos & Zeng, Ning, 2011. "Multivariate fractionally integrated APARCH modeling of stock market volatility: A multi-country study," Journal of Empirical Finance, Elsevier, vol. 18(1), pages 147-159, January.
    6. Wang, Yudong & Liu, Li & Ma, Feng & Wu, Chongfeng, 2016. "What the investors need to know about forecasting oil futures return volatility," Energy Economics, Elsevier, vol. 57(C), pages 128-139.
    7. Peter Reinhard Hansen & Asger Lunde & James M. Nason, 2003. "Choosing the Best Volatility Models: The Model Confidence Set Approach," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 839-861, December.
    8. Ma, Feng & Wei, Yu & Huang, Dengshi & Chen, Yixiang, 2014. "Which is the better forecasting model? A comparison between HAR-RV and multifractality volatility," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 405(C), pages 171-180.
    9. Amélie Charles & Olivier Darné, 2019. "The accuracy of asymmetric GARCH model estimation," International Economics, CEPII research center, issue 157, pages 179-202.
    10. Hansen, Peter Reinhard & Lunde, Asger, 2006. "Consistent ranking of volatility models," Journal of Econometrics, Elsevier, vol. 131(1-2), pages 97-121.
    11. Mauro Bernardi & Leopoldo Catania, 2016. "Comparison of Value-at-Risk models using the MCS approach," Computational Statistics, Springer, vol. 31(2), pages 579-608, June.
    12. Olivier Darné & Amélie Charles, 2020. "The accuracy of asymmetric GARCH model estimation," Working Papers hal-01943883, HAL.
    13. Awartani, Basel M.A. & Corradi, Valentina, 2005. "Predicting the volatility of the S&P-500 stock index via GARCH models: the role of asymmetries," International Journal of Forecasting, Elsevier, vol. 21(1), pages 167-183.
    14. Gavriilidis, Konstantinos & Kambouroudis, Dimos S. & Tsakou, Katerina & Tsouknidis, Dimitris A., 2018. "Volatility forecasting across tanker freight rates: The role of oil price shocks," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 118(C), pages 376-391.
    15. Turan Bali & Panayiotis Theodossiou, 2007. "A conditional-SGT-VaR approach with alternative GARCH models," Annals of Operations Research, Springer, vol. 151(1), pages 241-267, April.
    16. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra.
    17. Tae-Hwy Lee & Yong Bao & Burak Saltoglu, 2006. "Evaluating predictive performance of value-at-risk models in emerging markets: a reality check," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 25(2), pages 101-128.
    18. Franses,Philip Hans & Dijk,Dick van, 2000. "Non-Linear Time Series Models in Empirical Finance," Cambridge Books, Cambridge University Press, number 9780521779654, December.
    19. Bekaert, Geert & Engstrom, Eric & Ermolov, Andrey, 2015. "Bad environments, good environments: A non-Gaussian asymmetric volatility model," Journal of Econometrics, Elsevier, vol. 186(1), pages 258-275.
    20. Wei, Yu & Wang, Yudong & Huang, Dengshi, 2010. "Forecasting crude oil market volatility: Further evidence using GARCH-class models," Energy Economics, Elsevier, vol. 32(6), pages 1477-1484, November.
    21. esposito, francesco paolo & cummins, mark, 2015. "Multiple hypothesis testing of market risk forecasting models," MPRA Paper 64986, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jof:jforec:v:26:y:2007:i:3:p:203-225. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing) or (Christopher F. Baum). General contact details of provider: http://www3.interscience.wiley.com/cgi-bin/jhome/2966 .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.