IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v334y2024i1d10.1007_s10479-022-04611-9.html
   My bibliography  Save this article

Evaluating the discrimination ability of proper multi-variate scoring rules

Author

Listed:
  • C. Alexander

    (University of Sussex Business School)

  • M. Coulon

    (University of Sussex Business School)

  • Y. Han

    (University of Sussex Business School)

  • X. Meng

    (University of Sussex Business School)

Abstract

Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalized discrimination heuristic. Overall, we find that the variogram score with parameter $$p=0.5$$ p = 0.5 outperforms the energy score and the other two variogram scores.

Suggested Citation

  • C. Alexander & M. Coulon & Y. Han & X. Meng, 2024. "Evaluating the discrimination ability of proper multi-variate scoring rules," Annals of Operations Research, Springer, vol. 334(1), pages 857-883, March.
  • Handle: RePEc:spr:annopr:v:334:y:2024:i:1:d:10.1007_s10479-022-04611-9
    DOI: 10.1007/s10479-022-04611-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-022-04611-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s10479-022-04611-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael C. Jensen, 1968. "The Performance Of Mutual Funds In The Period 1945–1964," Journal of Finance, American Finance Association, vol. 23(2), pages 389-416, May.
    2. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    3. Y. Zhang & S. Nadarajah, 2018. "A review of backtesting for value at risk," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 47(15), pages 3616-3639, August.
    4. Danielsson, Jon & James, Kevin R. & Valenzuela, Marcela & Zer, Ilknur, 2016. "Model risk of risk models," Journal of Financial Stability, Elsevier, vol. 23(C), pages 79-91.
    5. Diks, Cees & Panchenko, Valentyn & Sokolinskiy, Oleg & van Dijk, Dick, 2014. "Comparing the accuracy of multivariate density forecasts in selected regions of the copula support," Journal of Economic Dynamics and Control, Elsevier, vol. 48(C), pages 79-94.
    6. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    7. Pérignon, Christophe & Smith, Daniel R., 2010. "The level and quality of Value-at-Risk disclosure by commercial banks," Journal of Banking & Finance, Elsevier, vol. 34(2), pages 362-377, February.
    8. Bollerslev, Tim, 1986. "Generalized autoregressive conditional heteroskedasticity," Journal of Econometrics, Elsevier, vol. 31(3), pages 307-327, April.
    9. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    10. Tilmann Gneiting & Roopesh Ranjan, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 411-422, July.
    11. Asger Lunde & Peter R. Hansen, 2005. "A forecast comparison of volatility models: does anything beat a GARCH(1,1)?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 20(7), pages 873-889.
    12. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    13. Bollerslev, Tim, 1990. "Modelling the Coherence in Short-run Nominal Exchange Rates: A Multivariate Generalized ARCH Model," The Review of Economics and Statistics, MIT Press, vol. 72(3), pages 498-505, August.
    14. Robert Engle, 2001. "GARCH 101: The Use of ARCH/GARCH Models in Applied Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 157-168, Fall.
    15. David J. Johnstone & Victor Richmond R. Jose & Robert L. Winkler, 2011. "Tailored Scoring Rules for Probabilities," Decision Analysis, INFORMS, vol. 8(4), pages 256-268, December.
    16. Florian Ziel & Kevin Berk, 2019. "Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules," Papers 1910.07325, arXiv.org.
    17. repec:hal:journl:peer-00834423 is not listed on IDEAS
    18. Alexander, Carol & Kaeck, Andreas & Sumawong, Anannit, 2019. "A parsimonious parametric model for generating margin requirements for futures," European Journal of Operational Research, Elsevier, vol. 273(1), pages 31-43.
    19. Engle, Robert F, 1982. "Autoregressive Conditional Heteroscedasticity with Estimates of the Variance of United Kingdom Inflation," Econometrica, Econometric Society, vol. 50(4), pages 987-1007, July.
    20. Nelson, Daniel B, 1991. "Conditional Heteroskedasticity in Asset Returns: A New Approach," Econometrica, Econometric Society, vol. 59(2), pages 347-370, March.
    21. Han Lin Shang & Yang Yang & Fearghal Kearney, 2019. "Intraday forecasts of a volatility index: functional time series methods with dynamic updating," Annals of Operations Research, Springer, vol. 282(1), pages 331-354, November.
    22. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    23. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.
    24. Anghel, Dan Gabriel, 2021. "Data Snooping Bias in Tests of the Relative Performance of Multiple Forecasting Models," Journal of Banking & Finance, Elsevier, vol. 126(C).
    25. Bauwens, Luc & Laurent, Sebastien, 2005. "A New Class of Multivariate Skew Densities, With Application to Generalized Autoregressive Conditional Heteroscedasticity Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 346-354, July.
    26. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    27. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    28. Pinson, P. & Girard, R., 2012. "Evaluating the quality of scenarios of short-term wind power generation," Applied Energy, Elsevier, vol. 96(C), pages 12-20.
    29. Engle, Robert, 2002. "Dynamic Conditional Correlation: A Simple Class of Multivariate Generalized Autoregressive Conditional Heteroskedasticity Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(3), pages 339-350, July.
    30. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    31. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    32. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    33. J. Eric Bickel, 2007. "Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules," Decision Analysis, INFORMS, vol. 4(2), pages 49-65, June.
    34. Gneiting, Tilmann & Ranjan, Roopesh, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(3), pages 411-422.
    35. Benoit Mandelbrot, 2015. "The Variation of Certain Speculative Prices," World Scientific Book Chapters, in: Anastasios G Malliaris & William T Ziemba (ed.), THE WORLD SCIENTIFIC HANDBOOK OF FUTURES MARKETS, chapter 3, pages 39-78, World Scientific Publishing Co. Pte. Ltd..
    36. Tsui, Albert K. & Yu, Qiao, 1999. "Constant conditional correlation in a bivariate GARCH model: evidence from the stock markets of China," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 48(4), pages 503-509.
    37. Filippo Curti & Ibrahim Ergen & Minh Le & Marco Migueis & Rob T. Stewart, 2016. "Benchmarking Operational Risk Models," Finance and Economics Discussion Series 2016-070, Board of Governors of the Federal Reserve System (U.S.).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Panagiotelis, Anastasios & Gamakumara, Puwasala & Athanasopoulos, George & Hyndman, Rob J., 2023. "Probabilistic forecast reconciliation: Properties, evaluation and score optimisation," European Journal of Operational Research, Elsevier, vol. 306(2), pages 693-706.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander, Carol & Han, Yang & Meng, Xiaochun, 2023. "Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile versus multivariate GARCH models," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1078-1096.
    2. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    3. Jie Cheng, 2024. "Evaluating Density Forecasts Using Weighted Multivariate Scores in a Risk Management Context," Computational Economics, Springer;Society for Computational Economics, vol. 64(6), pages 3617-3643, December.
    4. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    5. Hua, Jian & Manzan, Sebastiano, 2013. "Forecasting the return distribution using high-frequency volatility measures," Journal of Banking & Finance, Elsevier, vol. 37(11), pages 4381-4403.
    6. Ardia, David & Bluteau, Keven & Boudt, Kris & Catania, Leopoldo, 2018. "Forecasting risk with Markov-switching GARCH models:A large-scale performance study," International Journal of Forecasting, Elsevier, vol. 34(4), pages 733-747.
    7. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    8. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2005. "Volatility forecasting," CFS Working Paper Series 2005/08, Center for Financial Studies (CFS).
    9. Onno Kleen, 2024. "Scaling and measurement error sensitivity of scoring rules for distribution forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 833-849, August.
    10. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2006. "Volatility and Correlation Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 15, pages 777-878, Elsevier.
    11. Nikolaos A. Kyriazis, 2021. "A Survey on Volatility Fluctuations in the Decentralized Cryptocurrency Financial Assets," JRFM, MDPI, vol. 14(7), pages 1-46, June.
    12. Tae-Hwy Lee & Yong Bao & Burak Saltoğlu, 2007. "Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003;," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 203-225.
    13. Fabian Krüger & Sebastian Lerch & Thordis Thorarinsdottir & Tilmann Gneiting, 2021. "Predictive Inference Based on Markov Chain Monte Carlo Output," International Statistical Review, International Statistical Institute, vol. 89(2), pages 274-301, August.
    14. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    15. Ruili Sun & Tiefeng Ma & Shuangzhe Liu & Milind Sathye, 2019. "Improved Covariance Matrix Estimation for Portfolio Risk Measurement: A Review," JRFM, MDPI, vol. 12(1), pages 1-34, March.
    16. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    17. BAUWENS, Luc & HAFNER, Christian & LAURENT, Sébastien, 2011. "Volatility models," LIDAM Discussion Papers CORE 2011058, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
      • Bauwens, L. & Hafner C. & Laurent, S., 2011. "Volatility Models," LIDAM Discussion Papers ISBA 2011044, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
      • Bauwens, L. & Hafner, C. & Laurent, S., 2012. "Volatility Models," LIDAM Reprints ISBA 2012028, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    18. Carlos Henrique Dias Cordeiro de Castro & Fernando Antonio Lucena Aiube, 2023. "Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 369-401, March.
    19. Sébastien Laurent & Jeroen V. K. Rombouts & Francesco Violante, 2012. "On the forecasting accuracy of multivariate GARCH models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(6), pages 934-955, September.
    20. Wang, Yudong & Wu, Chongfeng, 2012. "Forecasting energy market volatility using GARCH models: Can multivariate models beat univariate models?," Energy Economics, Elsevier, vol. 34(6), pages 2167-2181.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:334:y:2024:i:1:d:10.1007_s10479-022-04611-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.