IDEAS home Printed from
   My bibliography  Save this paper

Robust Evaluation of Multivariate Density Forecasts


  • Dovern, Jonas
  • Manner, Hans


We derive new tests for proper calibration of multivariate density forecasts based on Rosenblatt probability integral transforms. These tests have the advantage that they i) do not depend on the ordering of variables in the forecasting model, ii) are applicable to densities of arbitrary dimensions, and iii) have superior power relative to existing approaches. We furthermore develop adjusted tests that allow for estimated parameters and, consequently, can be used as in-sample specification tests. We demonstrate the problems of existing tests and how our new approaches can overcome those using two applications based on multivariate GARCH-based models for stock market returns and on a macroeconomic Bayesian vectorautoregressive model.

Suggested Citation

  • Dovern, Jonas & Manner, Hans, 2016. "Robust Evaluation of Multivariate Density Forecasts," Annual Conference 2016 (Augsburg): Demographic Change 145547, Verein für Socialpolitik / German Economic Association.
  • Handle: RePEc:zbw:vfsc16:145547

    Download full text from publisher

    File URL:
    Download Restriction: no

    References listed on IDEAS

    1. James W. Taylor, 2012. "Density Forecasting of Intraday Call Center Arrivals Using Models Based on Exponential Smoothing," Management Science, INFORMS, vol. 58(3), pages 534-549, March.
    2. Isao Ishida, 2005. "Scanning Multivariate Conditional Densities with Probability Integral Transforms," CIRJE F-Series CIRJE-F-369, CIRJE, Faculty of Economics, University of Tokyo.
    3. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    4. Corradi, Valentina & Swanson, Norman R., 2006. "Bootstrap conditional distribution tests in the presence of dynamic misspecification," Journal of Econometrics, Elsevier, vol. 133(2), pages 779-806, August.
    5. Dovern, Jonas & Feldkircher, Martin & Huber, Florian, 2016. "Does joint modelling of the world economy pay off? Evaluating global forecasts from a Bayesian GVAR," Journal of Economic Dynamics and Control, Elsevier, vol. 70(C), pages 86-100.
    6. Yongmiao Hong, 2005. "Nonparametric Specification Testing for Continuous-Time Models with Applications to Term Structure of Interest Rates," Review of Financial Studies, Society for Financial Studies, vol. 18(1), pages 37-84.
    7. Maik H. Wolters, 2015. "Evaluating Point and Density Forecasts of DSGE Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(1), pages 74-96, January.
    8. Wolters, Maik H., 2011. "Forecasting under Model Uncertainty," Annual Conference 2011 (Frankfurt, Main): The Order of the World Economy - Lessons from the Crisis 48723, Verein für Socialpolitik / German Economic Association.
    9. Mark Hallam & Jose Olmo, 2014. "Semiparametric Density Forecasts of Daily Financial Returns from Intraday Data," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 12(2), pages 408-432.
    10. Shackleton, Mark B. & Taylor, Stephen J. & Yu, Peng, 2010. "A multi-horizon comparison of density forecasts for the S&P 500 using index returns and option prices," Journal of Banking & Finance, Elsevier, vol. 34(11), pages 2678-2693, November.
    11. Herbst, Edward & Schorfheide, Frank, 2012. "Evaluating DSGE model forecasts of comovements," Journal of Econometrics, Elsevier, vol. 171(2), pages 152-166.
    12. Knut Are Aastveit & Karsten R. Gerdrup & Anne Sofie Jore & Leif Anders Thorsrud, 2014. "Nowcasting GDP in Real Time: A Density Combination Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(1), pages 48-68, January.
    13. Glosten, Lawrence R & Jagannathan, Ravi & Runkle, David E, 1993. " On the Relation between the Expected Value and the Volatility of the Nominal Excess Return on Stocks," Journal of Finance, American Finance Association, vol. 48(5), pages 1779-1801, December.
    14. Jesús Crespo Cuaresma & Martin Feldkircher & Florian Huber, 2014. "Forecasting with Bayesian Global Vector Autoregressive Models: A Comparison of Priors," Working Papers 189, Oesterreichische Nationalbank (Austrian Central Bank).
    15. Todd E. Clark, 2011. "Real-Time Density Forecasts From Bayesian Vector Autoregressions With Stochastic Volatility," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 327-341, July.
    16. Hans Manner & Olga Reznikova, 2012. "A Survey on Time-Varying Copulas: Specification, Simulations, and Application," Econometric Reviews, Taylor & Francis Journals, vol. 31(6), pages 654-687, November.
    17. Clements, Michael P. & Smith, Jeremy, 2002. "Evaluating multivariate forecast densities: a comparison of two approaches," International Journal of Forecasting, Elsevier, vol. 18(3), pages 397-407.
    18. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 33(1), pages 125-132.
    19. Giorgio E. Primiceri, 2005. "Time Varying Structural Vector Autoregressions and Monetary Policy," Review of Economic Studies, Oxford University Press, vol. 72(3), pages 821-852.
    20. Huurman, Christian & Ravazzolo, Francesco & Zhou, Chen, 2012. "The power of weather," Computational Statistics & Data Analysis, Elsevier, vol. 56(11), pages 3793-3807.
    21. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    22. Ko, Stanley I.M. & Park, Sung Y., 2013. "Multivariate density forecast evaluation: A modified approach," International Journal of Forecasting, Elsevier, vol. 29(3), pages 431-441.
    23. Francis X. Diebold & Jinyong Hahn & Anthony S. Tay, 1999. "Multivariate Density Forecast Evaluation And Calibration In Financial Risk Management: High-Frequency Returns On Foreign Exchange," The Review of Economics and Statistics, MIT Press, vol. 81(4), pages 661-673, November.
    24. Jan G. De Gooijer, 2007. "Power of the Neyman Smooth Test for Evaluating Multivariate Forecast Densities," Journal of Applied Statistics, Taylor & Francis Journals, vol. 34(4), pages 371-381.
    25. Malin Adolfson & Jesper Linde & Mattias Villani, 2007. "Forecasting Performance of an Open Economy DSGE Model," Econometric Reviews, Taylor & Francis Journals, vol. 26(2-4), pages 289-328.
    26. Donald W. K. Andrews, 1997. "A Conditional Kolmogorov Test," Econometrica, Econometric Society, vol. 65(5), pages 1097-1128, September.
    27. José González-Barrios & Federico O’Reilly & Raúl Rueda, 2010. "Durbin’s random substitution and conditional Monte Carlo," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 72(3), pages 369-383, November.
    28. Bai, Jushan & Chen, Zhihong, 2008. "Testing multivariate distributions in GARCH models," Journal of Econometrics, Elsevier, vol. 143(1), pages 19-36, March.
    29. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    30. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    31. González-Rivera, Gloria & Yoldas, Emre, 2012. "Autocontour-based evaluation of multivariate predictive densities," International Journal of Forecasting, Elsevier, vol. 28(2), pages 328-342.
    32. Kitsul, Yuriy & Wright, Jonathan H., 2013. "The economics of options-implied inflation probability density functions," Journal of Financial Economics, Elsevier, vol. 110(3), pages 696-711.
    33. Hayfield, Tristen & Racine, Jeffrey S., 2008. "Nonparametric Econometrics: The np Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 27(i05).
    34. Qi Li & Juan Lin & Jeffrey S. Racine, 2013. "Optimal Bandwidth Selection for Nonparametric Conditional Distribution and Quantile Functions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(1), pages 57-65, January.
    35. Engle, Robert, 2002. "Dynamic Conditional Correlation: A Simple Class of Multivariate Generalized Autoregressive Conditional Heteroskedasticity Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(3), pages 339-350, July.
    36. Hansen, Bruce E, 1994. "Autoregressive Conditional Density Estimation," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 35(3), pages 705-730, August.
    37. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268.
    38. Joe, Harry, 2005. "Asymptotic efficiency of the two-stage estimation method for copula-based models," Journal of Multivariate Analysis, Elsevier, vol. 94(2), pages 401-419, June.
    39. Jushan Bai, 2003. "Testing Parametric Conditional Distributions of Dynamic Models," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 531-549, August.
    40. James Mitchell & Kenneth F. Wallis, 2011. "Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(6), pages 1023-1040, September.
    41. Genest, Christian & Rivest, Louis-Paul, 2001. "On the multivariate probability integral transformation," Statistics & Probability Letters, Elsevier, vol. 53(4), pages 391-399, July.
    Full references (including those not matched with items on IDEAS)

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:vfsc16:145547. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (ZBW - German National Library of Economics). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.