IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2002.09578.html
   My bibliography  Save this paper

Scores for Multivariate Distributions and Level Sets

Author

Listed:
  • Xiaochun Meng
  • James W. Taylor
  • Souhaib Ben Taieb
  • Siran Li

Abstract

Forecasts of multivariate probability distributions are required for a variety of applications. Scoring rules enable the evaluation of forecast accuracy, and comparison between forecasting methods. We propose a theoretical framework for scoring rules for multivariate distributions, which encompasses the existing quadratic score and multivariate continuous ranked probability score. We demonstrate how this framework can be used to generate new scoring rules. In some multivariate contexts, it is a forecast of a level set that is needed, such as a density level set for anomaly detection or the level set of the cumulative distribution as a measure of risk. This motivates consideration of scoring functions for such level sets. For univariate distributions, it is well-established that the continuous ranked probability score can be expressed as the integral over a quantile score. We show that, in a similar way, scoring rules for multivariate distributions can be decomposed to obtain scoring functions for level sets. Using this, we present scoring functions for different types of level set, including density level sets and level sets for cumulative distributions. To compute the scores, we propose a simple numerical algorithm. We perform a simulation study to support our proposals, and we use real data to illustrate usefulness for forecast combining and CoVaR estimation.

Suggested Citation

  • Xiaochun Meng & James W. Taylor & Souhaib Ben Taieb & Siran Li, 2020. "Scores for Multivariate Distributions and Level Sets," Papers 2002.09578, arXiv.org, revised Jun 2023.
  • Handle: RePEc:arx:papers:2002.09578
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2002.09578
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Komunjer, Ivana, 2005. "Quasi-maximum likelihood estimation for conditional quantiles," Journal of Econometrics, Elsevier, vol. 128(1), pages 137-164, September.
    2. Yen-Chi Chen & Christopher R. Genovese & Larry Wasserman, 2017. "Density Level Sets: Asymptotics, Inference, and Visualization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1684-1696, October.
    3. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    4. Anthonisz, Sean A., 2012. "Asset pricing with partial-moments," Journal of Banking & Finance, Elsevier, vol. 36(7), pages 2122-2135.
    5. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    6. Ali E. Abbas & David V. Budescu & Yuhong (Rola) Gu, 2010. "Assessing Joint Distributions with Isoprobability Contours," Management Science, INFORMS, vol. 56(6), pages 997-1011, June.
    7. Koenker, Roger W & Bassett, Gilbert, Jr, 1978. "Regression Quantiles," Econometrica, Econometric Society, vol. 46(1), pages 33-50, January.
    8. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    9. Patton, Andrew J., 2012. "A review of copula models for economic time series," Journal of Multivariate Analysis, Elsevier, vol. 110(C), pages 4-18.
    10. Tilmann Gneiting & Roopesh Ranjan, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 411-422, July.
    11. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    12. Areski Cousin & Elena Di Bernadino, 2013. "On Multivariate Extensions of Value-at-Risk," Working Papers hal-00638382, HAL.
    13. Areski Cousin & Elena Di Bernadino, 2011. "On Multivariate Extensions of Value-at-Risk," Papers 1111.1349, arXiv.org, revised Apr 2013.
    14. Robert F. Engle & Simone Manganelli, 2004. "CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles," Journal of Business & Economic Statistics, American Statistical Association, vol. 22, pages 367-381, October.
    15. Diks, Cees & Panchenko, Valentyn & Sokolinskiy, Oleg & van Dijk, Dick, 2014. "Comparing the accuracy of multivariate density forecasts in selected regions of the copula support," Journal of Economic Dynamics and Control, Elsevier, vol. 48(C), pages 79-94.
    16. Jooyoung Jeon & James W. Taylor, 2012. "Using Conditional Kernel Density Estimation for Wind Power Density Forecasting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 66-79, March.
    17. Elena Di Bernardino & Thomas Laloë & Véronique Maume-Deschamps & Clémentine Prieur, 2013. "Plug-in estimation of level sets in a non-compact setting with applications in multivariate risk theory," Post-Print hal-00580624, HAL.
    18. Marc Hallin & Davy Paindaveine & Miroslav Siman, 2008. "Multivariate quantiles and multiple-output regression quantiles: from L1 optimization to halfspace depth," Working Papers ECARES 2008_042, ULB -- Universite Libre de Bruxelles.
    19. Cousin, Areski & Di Bernardino, Elena, 2013. "On multivariate extensions of Value-at-Risk," Journal of Multivariate Analysis, Elsevier, vol. 119(C), pages 32-46.
    20. Gneiting, Tilmann & Ranjan, Roopesh, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(3), pages 411-422.
    21. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    22. Cadre, BenoI^t, 2006. "Kernel estimation of density level sets," Journal of Multivariate Analysis, Elsevier, vol. 97(4), pages 999-1023, April.
    23. Briec, Walter & Kerstens, Kristiaan, 2010. "Portfolio selection in multidimensional general and partial moment space," Journal of Economic Dynamics and Control, Elsevier, vol. 34(4), pages 636-656, April.
    24. M.‐Y. Cheng & P. Hall, 1998. "Calibrating the excess mass and dip tests of modality," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 60(3), pages 579-589.
    25. Price, Kelly & Price, Barbara & Nantell, Timothy J, 1982. "Variance and Lower Partial Moment Measures of Systematic Risk: Some Analytical and Empirical Results," Journal of Finance, American Finance Association, vol. 37(3), pages 843-855, June.
    26. repec:hal:journl:peer-00834423 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    2. Klaus Herrmann & Marius Hofert & Melina Mailhot, 2017. "Multivariate Geometric Expectiles," Papers 1704.01503, arXiv.org, revised Jan 2018.
    3. Alexander, Carol & Han, Yang & Meng, Xiaochun, 2023. "Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile versus multivariate GARCH models," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1078-1096.
    4. Bruzda, Joanna, 2019. "Quantile smoothing in supply chain and logistics forecasting," International Journal of Production Economics, Elsevier, vol. 208(C), pages 122-139.
    5. Yuru Sun & Worapree Maneesoonthorn & Ruben Loaiza-Maya & Gael M. Martin, 2023. "Optimal probabilistic forecasts for risk management," Papers 2303.01651, arXiv.org.
    6. Ruben Loaiza‐Maya & Gael M. Martin & David T. Frazier, 2021. "Focused Bayesian prediction," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(5), pages 517-543, August.
    7. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    8. Catania, Leopoldo & Grassi, Stefano, 2022. "Forecasting cryptocurrency volatility," International Journal of Forecasting, Elsevier, vol. 38(3), pages 878-894.
    9. Timo Dimitriadis & Tobias Fissler & Johanna Ziegel, 2022. "Characterizing M-estimators," Papers 2208.08108, arXiv.org.
    10. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    11. Jonas R. Brehmer & Tilmann Gneiting, 2020. "Properization: constructing proper scoring rules via Bayes acts," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(3), pages 659-673, June.
    12. Tobias Fissler & Hajo Holzmann, 2022. "Measurability of functionals and of ideal point forecasts," Papers 2203.08635, arXiv.org.
    13. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    14. Timo Dimitriadis & Tobias Fissler & Johanna Ziegel, 2020. "The Efficiency Gap," Papers 2010.14146, arXiv.org, revised Sep 2022.
    15. Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose & Robert L. Winkler, 2017. "Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs," Operations Research, INFORMS, vol. 65(3), pages 712-728, June.
    16. Taillardat, Maxime & Fougères, Anne-Laure & Naveau, Philippe & de Fondeville, Raphaël, 2023. "Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1448-1459.
    17. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    18. Alexander Henzi & Johanna F. Ziegel & Tilmann Gneiting, 2021. "Isotonic distributional regression," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 963-993, November.
    19. Müller, Alfred & Reuber, Matthias, 2023. "A copula-based time series model for global horizontal irradiation," International Journal of Forecasting, Elsevier, vol. 39(2), pages 869-883.
    20. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2002.09578. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.