IDEAS home Printed from https://ideas.repec.org/a/jof/jforec/v26y2007i3p203-225.html
   My bibliography  Save this article

Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003; see Bao et al. (2004)

Author

Listed:
  • Tae-Hwy Lee

    (University of California, Riverside, California USA)

  • Yong Bao

    (University of Texas, San Antonio, Texas, USA)

  • Burak Saltoğlu

    (Bosphorous University, Istanbul, Turkey)

Abstract

In this paper we discuss how to compare various (possibly misspecified) density forecast models using the Kullback-Leibler information criterion (KLIC) of a candidate density forecast model with respect to the true density. The KLIC differential between a pair of competing models is the (predictive) log-likelihood ratio (LR) between the two models. Even though the true density is unknown, using the LR statistic amounts to comparing models with the KLIC as a loss function and thus enables us to assess which density forecast model can approximate the true density more closely. We also discuss how this KLIC is related to the KLIC based on the probability integral transform (PIT) in the framework of Diebold et al. (1998). While they are asymptotically equivalent, the PIT-based KLIC is best suited for evaluating the adequacy of each density forecast model and the original KLIC is best suited for comparing competing models. In an empirical study with the S&P500 and NASDAQ daily return series, we find strong evidence for rejecting the normal-GARCH benchmark model, in favor of the models that can capture skewness in the conditional distribution and asymmetry and long memory in the conditional variance. Copyright © 2007 John Wiley & Sons, Ltd.

Suggested Citation

  • Tae-Hwy Lee & Yong Bao & Burak Saltoğlu, 2007. "Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003;," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 203-225.
  • Handle: RePEc:jof:jforec:v:26:y:2007:i:3:p:203-225
    DOI: 10.1002/for.1023
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/for.1023
    File Function: Link to full text; subscription required
    Download Restriction: no

    File URL: https://libkey.io/10.1002/for.1023?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jondeau, Eric & Rockinger, Michael, 2001. "Gram-Charlier densities," Journal of Economic Dynamics and Control, Elsevier, vol. 25(10), pages 1457-1483, October.
    2. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    3. Nelson, Daniel B, 1991. "Conditional Heteroskedasticity in Asset Returns: A New Approach," Econometrica, Econometric Society, vol. 59(2), pages 347-370, March.
    4. Panayiotis Theodossiou, 1998. "Financial Data and the Skewed Generalized T Distribution," Management Science, INFORMS, vol. 44(12-Part-1), pages 1650-1661, December.
    5. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    6. Ding, Zhuanxin & Granger, Clive W. J. & Engle, Robert F., 1993. "A long memory property of stock market returns and a new model," Journal of Empirical Finance, Elsevier, vol. 1(1), pages 83-106, June.
    7. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    8. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    9. Bollerslev, Tim & Ole Mikkelsen, Hans, 1996. "Modeling and pricing long memory in stock market volatility," Journal of Econometrics, Elsevier, vol. 73(1), pages 151-184, July.
    10. Torben G. Andersen & Luca Benzoni & Jesper Lund, 2002. "An Empirical Investigation of Continuous‐Time Equity Return Models," Journal of Finance, American Finance Association, vol. 57(3), pages 1239-1284, June.
    11. White,Halbert, 1996. "Estimation, Inference and Specification Analysis," Cambridge Books, Cambridge University Press, number 9780521574464, January.
    12. Bauwens, Luc & Giot, Pierre & Grammig, Joachim & Veredas, David, 2004. "A comparison of financial duration models via density forecasts," International Journal of Forecasting, Elsevier, vol. 20(4), pages 589-609.
    13. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    14. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    15. Kandel, Shmuel & Stambaugh, Robert F, 1996. "On the Predictability of Stock Returns: An Asset-Allocation Perspective," Journal of Finance, American Finance Association, vol. 51(2), pages 385-424, June.
    16. Dr. James Mitchell, 2005. "Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ÔfanÕ charts of inflation," National Institute of Economic and Social Research (NIESR) Discussion Papers 253, National Institute of Economic and Social Research.
    17. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    18. Yihong Xia, 2001. "Learning about Predictability: The Effects of Parameter Uncertainty on Dynamic Asset Allocation," Journal of Finance, American Finance Association, vol. 56(1), pages 205-246, February.
    19. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    20. Gallant, A Ronald & Nychka, Douglas W, 1987. "Semi-nonparametric Maximum Likelihood Estimation," Econometrica, Econometric Society, vol. 55(2), pages 363-390, March.
    21. West, Kenneth D & McCracken, Michael W, 1998. "Regression-Based Tests of Predictive Ability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 817-840, November.
    22. Giorgio Valente & Lucio Sarno, 2004. "Comparing the accuracy of density forecasts from competing models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(8), pages 541-557.
    23. Jackwerth, Jens Carsten & Rubinstein, Mark, 1996. "Recovering Probability Distributions from Option Prices," Journal of Finance, American Finance Association, vol. 51(5), pages 1611-1632, December.
    24. James Mitchell & Stephen G. Hall, 2005. "Evaluating, Comparing and Combining Density Forecasts Using the KLIC with an Application to the Bank of England and NIESR ‘Fan’ Charts of Inflation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 67(s1), pages 995-1033, December.
    25. Corradi, Valentina & Swanson, Norman R., 2006. "Predictive density and conditional confidence interval accuracy tests," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 187-228.
    26. Subu Venkataraman, 1997. "Value at risk for a mixture of normal distributions: the use of quasi- Bayesian estimation techniques," Economic Perspectives, Federal Reserve Bank of Chicago, vol. 21(Mar), pages 2-13.
    27. Zakoian, Jean-Michel, 1994. "Threshold heteroskedastic models," Journal of Economic Dynamics and Control, Elsevier, vol. 18(5), pages 931-955, September.
    28. Glosten, Lawrence R & Jagannathan, Ravi & Runkle, David E, 1993. "On the Relation between the Expected Value and the Volatility of the Nominal Excess Return on Stocks," Journal of Finance, American Finance Association, vol. 48(5), pages 1779-1801, December.
    29. McDonald, James B. & Xu, Yexiao J., 1995. "A generalization of the beta distribution with applications," Journal of Econometrics, Elsevier, vol. 69(2), pages 427-428, October.
    30. Bollerslev, Tim, 1986. "Generalized autoregressive conditional heteroskedasticity," Journal of Econometrics, Elsevier, vol. 31(3), pages 307-327, April.
    31. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    32. Christian Gourieroux & Joann Jasiak, 2010. "Local Likelihood Density Estimation and Value-at-Risk," Journal of Probability and Statistics, Hindawi, vol. 2010, pages 1-26, June.
    33. Ser-Huang Poon & Clive W.J. Granger, 2003. "Forecasting Volatility in Financial Markets: A Review," Journal of Economic Literature, American Economic Association, vol. 41(2), pages 478-539, June.
    34. Pascual, Lorenzo & Romo, Juan & Ruiz, Esther, 2001. "Effects of parameter estimation on prediction densities: a bootstrap approach," International Journal of Forecasting, Elsevier, vol. 17(1), pages 83-103.
    35. Ding, Zhuanxin & Granger, Clive W. J., 1996. "Modeling volatility persistence of speculative returns: A new approach," Journal of Econometrics, Elsevier, vol. 73(1), pages 185-215, July.
    36. Davidson, James, 2004. "Moment and Memory Properties of Linear Conditional Heteroscedasticity Models, and a New Model," Journal of Business & Economic Statistics, American Statistical Association, vol. 22(1), pages 16-29, January.
    37. Hong, Yongmiao & Lee, Tae-Hwy, 2003. "Diagnostic Checking For The Adequacy Of Nonlinear Time Series Models," Econometric Theory, Cambridge University Press, vol. 19(6), pages 1065-1121, December.
    38. J. L. Knight & S. E. Satchell & K. C. Tran, 1995. "Statistical modelling of asymmetric risk in asset returns," Applied Mathematical Finance, Taylor & Francis Journals, vol. 2(3), pages 155-172.
    39. Engle, Robert F & Gonzalez-Rivera, Gloria, 1991. "Semiparametric ARCH Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 9(4), pages 345-359, October.
    40. Dr. James Mitchell, 2005. "Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ÔfanÕ charts of inflation," National Institute of Economic and Social Research (NIESR) Discussion Papers 253, National Institute of Economic and Social Research.
    41. McDonald, James B. & Newey, Whitney K., 1988. "Partially Adaptive Estimation of Regression Models via the Generalized T Distribution," Econometric Theory, Cambridge University Press, vol. 4(3), pages 428-457, December.
    42. Nicholas Barberis, 2000. "Investing for the Long Run when Returns Are Predictable," Journal of Finance, American Finance Association, vol. 55(1), pages 225-264, February.
    43. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    44. Sawa, Takamitsu, 1978. "Information Criteria for Discriminating among Alternative Regression Models," Econometrica, Econometric Society, vol. 46(6), pages 1273-1291, November.
    45. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lee, Tae-Hwy & Long, Xiangdong, 2009. "Copula-based multivariate GARCH model with uncorrelated dependent errors," Journal of Econometrics, Elsevier, vol. 150(2), pages 207-218, June.
    2. Ravazzolo Francesco & Vahey Shaun P., 2014. "Forecast densities for economic aggregates from disaggregate ensembles," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 18(4), pages 1-15, September.
    3. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2010. "Out-of-sample comparison of copula specifications in multivariate density forecasts," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1596-1609, September.
    4. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    5. Maheu, John M. & McCurdy, Thomas H., 2011. "Do high-frequency measures of volatility improve forecasts of return distributions?," Journal of Econometrics, Elsevier, vol. 160(1), pages 69-76, January.
    6. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    7. Cheng, Xixin & Li, W.K. & Yu, Philip L.H. & Zhou, Xuan & Wang, Chao & Lo, P.H., 2011. "Modeling threshold conditional heteroscedasticity with regime-dependent skewness and kurtosis," Computational Statistics & Data Analysis, Elsevier, vol. 55(9), pages 2590-2604, September.
    8. Garratt, Anthony & Mitchell, James & Vahey, Shaun P., 2014. "Measuring output gap nowcast uncertainty," International Journal of Forecasting, Elsevier, vol. 30(2), pages 268-279.
    9. Li, Yushu & Andersson, Jonas, 2014. "A Likelihood Ratio and Markov Chain Based Method to Evaluate Density Forecasting," Discussion Papers 2014/12, Norwegian School of Economics, Department of Business and Management Science.
    10. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    11. Rompolis, Leonidas S., 2010. "Retrieving risk neutral densities from European option prices based on the principle of maximum entropy," Journal of Empirical Finance, Elsevier, vol. 17(5), pages 918-937, December.
    12. Hua, Jian & Manzan, Sebastiano, 2013. "Forecasting the return distribution using high-frequency volatility measures," Journal of Banking & Finance, Elsevier, vol. 37(11), pages 4381-4403.
    13. Del Brio, Esther B. & Ñíguez, Trino-Manuel & Perote, Javier, 2011. "Multivariate semi-nonparametric distributions with dynamic conditional correlations," International Journal of Forecasting, Elsevier, vol. 27(2), pages 347-364.
    14. Anthony Garratt & James Mitchell & Shaun P. Vahey, 2009. "Measuring Output Gap Uncertainty," Birkbeck Working Papers in Economics and Finance 0909, Birkbeck, Department of Economics, Mathematics & Statistics.
    15. Gloria González-Rivera & Tae-Hwy Lee, 2007. "Nonlinear Time Series in Financial Forecasting," Working Papers 200803, University of California at Riverside, Department of Economics, revised Feb 2008.
    16. Francesco Ravazzolo & Shaun P Vahey, 2010. "Measuring Core Inflation in Australia with Disaggregate Ensembles," RBA Annual Conference Volume (Discontinued), in: Renée Fry & Callum Jones & Christopher Kent (ed.),Inflation in an Era of Relative Price Shocks, Reserve Bank of Australia.
    17. Kyungchul Song, 2009. "Testing Predictive Ability and Power Robustification," PIER Working Paper Archive 09-035, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2006. "Volatility and Correlation Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 15, pages 777-878, Elsevier.
    2. Torben G. Andersen & Tim Bollerslev & Peter F. Christoffersen & Francis X. Diebold, 2005. "Volatility Forecasting," PIER Working Paper Archive 05-011, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    5. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    6. Wang, Yudong & Liu, Li & Ma, Feng & Wu, Chongfeng, 2016. "What the investors need to know about forecasting oil futures return volatility," Energy Economics, Elsevier, vol. 57(C), pages 128-139.
    7. Mehmet Sahiner, 2022. "Forecasting volatility in Asian financial markets: evidence from recursive and rolling window methods," SN Business & Economics, Springer, vol. 2(10), pages 1-74, October.
    8. Cees Diks & Valentyn Panchenko & Dick van Dijk, 2008. "Partial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails," Tinbergen Institute Discussion Papers 08-050/4, Tinbergen Institute.
    9. Meade, Nigel, 2010. "Oil prices -- Brownian motion or mean reversion? A study using a one year ahead density forecast criterion," Energy Economics, Elsevier, vol. 32(6), pages 1485-1498, November.
    10. Hansen, Peter Reinhard & Lunde, Asger, 2006. "Consistent ranking of volatility models," Journal of Econometrics, Elsevier, vol. 131(1-2), pages 97-121.
    11. Mauro Bernardi & Leopoldo Catania, 2016. "Comparison of Value-at-Risk models using the MCS approach," Computational Statistics, Springer, vol. 31(2), pages 579-608, June.
    12. Halkos, George E. & Tsirivis, Apostolos S., 2019. "Effective energy commodity risk management: Econometric modeling of price volatility," Economic Analysis and Policy, Elsevier, vol. 63(C), pages 234-250.
    13. Tae-Hwy Lee & Yong Bao & Burak Saltoglu, 2006. "Evaluating predictive performance of value-at-risk models in emerging markets: a reality check," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 25(2), pages 101-128.
    14. repec:awi:wpaper:0472 is not listed on IDEAS
    15. Peter Hansen & Asger Lunde & James M. Nason, 2003. "Choosing the Best Volatility Models:The Model Confidence Set Approach," Working Papers 2003-05, Brown University, Department of Economics.
    16. Wei, Yu & Wang, Yudong & Huang, Dengshi, 2010. "Forecasting crude oil market volatility: Further evidence using GARCH-class models," Energy Economics, Elsevier, vol. 32(6), pages 1477-1484, November.
    17. Ma, Feng & Wei, Yu & Huang, Dengshi & Chen, Yixiang, 2014. "Which is the better forecasting model? A comparison between HAR-RV and multifractality volatility," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 405(C), pages 171-180.
    18. repec:hal:journl:peer-00834423 is not listed on IDEAS
    19. Gloria González-Rivera & Tae-Hwy Lee, 2007. "Nonlinear Time Series in Financial Forecasting," Working Papers 200803, University of California at Riverside, Department of Economics, revised Feb 2008.
    20. Amélie Charles & Olivier Darné, 2019. "The accuracy of asymmetric GARCH model estimation," International Economics, CEPII research center, issue 157, pages 179-202.
    21. Shamiri, Ahmed & Shaari, Abu Hassan & Isa, Zaidi, 2008. "Comparing the accuracy of density forecasts from competing GARCH models," MPRA Paper 13662, University Library of Munich, Germany.
    22. Owusu Junior, Peterson & Tiwari, Aviral Kumar & Tweneboah, George & Asafo-Adjei, Emmanuel, 2022. "GAS and GARCH based value-at-risk modeling of precious metals," Resources Policy, Elsevier, vol. 75(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jof:jforec:v:26:y:2007:i:3:p:203-225. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley-Blackwell Digital Licensing or Christopher F. Baum (email available below). General contact details of provider: http://www3.interscience.wiley.com/cgi-bin/jhome/2966 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.