IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v307y2023i3p1408-1423.html
   My bibliography  Save this article

Sensitivity measures based on scoring functions

Author

Listed:
  • Fissler, Tobias
  • Pesenti, Silvana M.

Abstract

We propose a holistic framework for constructing sensitivity measures for any elicitable functional T of a response variable. The sensitivity measures, termed score-based sensitivities, are constructed via scoring functions that are (strictly) consistent for T. These score-based sensitivities quantify the relative improvement in predictive accuracy when available information, e.g., from explanatory variables, is used ideally. We establish intuitive and desirable properties of these sensitivities and discuss advantageous choices of scoring functions leading to scale-invariant sensitivities.

Suggested Citation

  • Fissler, Tobias & Pesenti, Silvana M., 2023. "Sensitivity measures based on scoring functions," European Journal of Operational Research, Elsevier, vol. 307(3), pages 1408-1423.
    • Handle: RePEc:eee:ejores:v:307:y:2023:i:3:p:1408-1423
    DOI: 10.1016/j.ejor.2022.10.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221722007718
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2022.10.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    2. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    3. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.
    4. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    5. Emanuele Borgonovo, 2017. "Value of Information," International Series in Operations Research & Management Science, in: Sensitivity Analysis, chapter 0, pages 93-100, Springer.
    6. Borgonovo, E., 2007. "A new uncertainty importance measure," Reliability Engineering and System Safety, Elsevier, vol. 92(6), pages 771-784.
    7. Silvana M. Pesenti, 2022. "Reverse Sensitivity Analysis for Risk Modelling," Risks, MDPI, vol. 10(7), pages 1-23, July.
    8. Patton, Andrew J., 2011. "Data-based ranking of realised volatility estimators," Journal of Econometrics, Elsevier, vol. 161(2), pages 284-303, April.
    9. Marrel, Amandine & Iooss, Bertrand & Laurent, Béatrice & Roustant, Olivier, 2009. "Calculations of Sobol indices for the Gaussian process metamodel," Reliability Engineering and System Safety, Elsevier, vol. 94(3), pages 742-751.
    10. Silvana M. Pesenti, 2021. "Reverse Sensitivity Analysis for Risk Modelling," Papers 2107.01065, arXiv.org, revised May 2022.
    11. Pesenti, Silvana M. & Millossovich, Pietro & Tsanakas, Andreas, 2019. "Reverse sensitivity testing: What does it take to break the model?," European Journal of Operational Research, Elsevier, vol. 274(2), pages 654-670.
    12. Plischke, Elmar & Borgonovo, Emanuele, 2019. "Copula theory and probabilistic sensitivity analysis: Is there a connection?," European Journal of Operational Research, Elsevier, vol. 277(3), pages 1046-1059.
    13. Rama Cont & Romain Deguest & Giacomo Scandolo, 2010. "Robustness and sensitivity analysis of risk measurement procedures," Post-Print hal-00413729, HAL.
    14. Manel Baucells & Emanuele Borgonovo, 2013. "Invariant Probabilistic Sensitivity Analysis," Management Science, INFORMS, vol. 59(11), pages 2536-2549, November.
    15. James C. Felli & Gordon B. Hazen, 1998. "Sensitivity Analysis and the Expected Value of Perfect Information," Medical Decision Making, , vol. 18(1), pages 95-109, January.
    16. Tobias Fissler & Hajo Holzmann, 2022. "Measurability of functionals and of ideal point forecasts," Papers 2203.08635, arXiv.org.
    17. Andreas Tsanakas & Pietro Millossovich, 2016. "Sensitivity Analysis Using Risk Measures," Risk Analysis, John Wiley & Sons, vol. 36(1), pages 30-48, January.
    18. Alexander Frankel & Emir Kamenica, 2019. "Quantifying Information and Uncertainty," American Economic Review, American Economic Association, vol. 109(10), pages 3650-3680, October.
    19. Saltelli A. & Tarantola S., 2002. "On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 702-709, September.
    20. Borgonovo, Emanuele & Plischke, Elmar, 2016. "Sensitivity analysis: A review of recent advances," European Journal of Operational Research, Elsevier, vol. 248(3), pages 869-887.
    21. Fabian Krüger & Johanna F. Ziegel, 2021. "Generic Conditions for Forecast Dominance," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(4), pages 972-983, October.
    22. Acerbi, Carlo, 2002. "Spectral measures of risk: A coherent representation of subjective risk aversion," Journal of Banking & Finance, Elsevier, vol. 26(7), pages 1505-1518, July.
    23. Alexander J. McNeil & Rüdiger Frey & Paul Embrechts, 2015. "Quantitative Risk Management: Concepts, Techniques and Tools Revised edition," Economics Books, Princeton University Press, edition 2, number 10496.
    24. Jean-Claude Fort & Thierry Klein & Nabil Rachdi, 2016. "New sensitivity analysis subordinated to a contrast," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 45(15), pages 4349-4364, August.
    25. Rama Cont & Romain Deguest & Giacomo Scandolo, 2010. "Robustness and sensitivity analysis of risk measurement procedures," Quantitative Finance, Taylor & Francis Journals, vol. 10(6), pages 593-606.
    26. Tobias Fissler & Michael Merz & Mario V. Wuthrich, 2021. "Deep Quantile and Deep Composite Model Regression," Papers 2112.03075, arXiv.org.
    27. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    28. Ruodu Wang & Yunran Wei, 2020. "Risk functionals with convex level sets," Mathematical Finance, Wiley Blackwell, vol. 30(4), pages 1337-1367, October.
    29. Gerber, Hans U., 1974. "On Additive Premium Calculation Principles," ASTIN Bulletin, Cambridge University Press, vol. 7(3), pages 215-222, March.
    30. Borgonovo, Emanuele & Hazen, Gordon B. & Jose, Victor Richmond R. & Plischke, Elmar, 2021. "Probabilistic sensitivity measures as information value," European Journal of Operational Research, Elsevier, vol. 289(2), pages 595-610.
    31. Stefan Weber, 2006. "Distribution‐Invariant Risk Measures, Information, And Dynamic Consistency," Mathematical Finance, Wiley Blackwell, vol. 16(2), pages 419-441, April.
    32. C. Heinrich, 2014. "The mode functional is not elicitable," Biometrika, Biometrika Trust, vol. 101(1), pages 245-251.
    33. Emanuele Borgonovo & Alessandra Cillo, 2017. "Deciding with Thresholds: Importance Measures and Value of Information," Risk Analysis, John Wiley & Sons, vol. 37(10), pages 1828-1848, October.
    34. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    35. Maume-Deschamps, Véronique & Niang, Ibrahima, 2018. "Estimation of quantile oriented sensitivity indices," Statistics & Probability Letters, Elsevier, vol. 134(C), pages 122-127.
    36. A. Dawid, 2007. "The geometry of proper scoring rules," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 59(1), pages 77-93, March.
    37. Hajo Holzmann & Matthias Eulert, 2014. "The role of the information set for forecasting - with applications to risk management," Papers 1404.7653, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Silvana M. Pesenti & Pietro Millossovich & Andreas Tsanakas, 2023. "Differential Sensitivity in Discontinuous Models," Papers 2310.06151, arXiv.org.
    2. Silvana M. Pesenti & Steven Vanduffel, 2023. "Optimal Transport Divergences induced by Scoring Functions," Papers 2311.12183, arXiv.org, revised Dec 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tobias Fissler & Silvana M. Pesenti, 2022. "Sensitivity Measures Based on Scoring Functions," Papers 2203.00460, arXiv.org, revised Jul 2022.
    2. Fissler Tobias & Ziegel Johanna F., 2021. "On the elicitability of range value at risk," Statistics & Risk Modeling, De Gruyter, vol. 38(1-2), pages 25-46, January.
    3. Borgonovo, Emanuele & Hazen, Gordon B. & Jose, Victor Richmond R. & Plischke, Elmar, 2021. "Probabilistic sensitivity measures as information value," European Journal of Operational Research, Elsevier, vol. 289(2), pages 595-610.
    4. Silvana M. Pesenti, 2021. "Reverse Sensitivity Analysis for Risk Modelling," Papers 2107.01065, arXiv.org, revised May 2022.
    5. Tobias Fissler & Hajo Holzmann, 2022. "Measurability of functionals and of ideal point forecasts," Papers 2203.08635, arXiv.org.
    6. Ruodu Wang & Yunran Wei, 2020. "Risk functionals with convex level sets," Mathematical Finance, Wiley Blackwell, vol. 30(4), pages 1337-1367, October.
    7. Tobias Fissler & Jana Hlavinová & Birgit Rudloff, 2021. "Elicitability and identifiability of set-valued measures of systemic risk," Finance and Stochastics, Springer, vol. 25(1), pages 133-165, January.
    8. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    9. Tobias Fissler & Johanna F. Ziegel, 2019. "Evaluating Range Value at Risk Forecasts," Papers 1902.04489, arXiv.org, revised Nov 2020.
    10. Steven Kou & Xianhua Peng, 2016. "On the Measurement of Economic Tail Risk," Operations Research, INFORMS, vol. 64(5), pages 1056-1072, October.
    11. Pesenti, Silvana M. & Millossovich, Pietro & Tsanakas, Andreas, 2019. "Reverse sensitivity testing: What does it take to break the model?," European Journal of Operational Research, Elsevier, vol. 274(2), pages 654-670.
    12. Makam, Vaishno Devi & Millossovich, Pietro & Tsanakas, Andreas, 2021. "Sensitivity analysis with χ2-divergences," Insurance: Mathematics and Economics, Elsevier, vol. 100(C), pages 372-383.
    13. Pitera, Marcin & Schmidt, Thorsten, 2018. "Unbiased estimation of risk," Journal of Banking & Finance, Elsevier, vol. 91(C), pages 133-145.
    14. Mucahit Aygun & Fabio Bellini & Roger J. A. Laeven, 2023. "Elicitability of Return Risk Measures," Papers 2302.13070, arXiv.org, revised Mar 2023.
    15. Zdeněk Kala, 2021. "New Importance Measures Based on Failure Probability in Global Sensitivity Analysis of Reliability," Mathematics, MDPI, vol. 9(19), pages 1-20, September.
    16. Fissler, Tobias & Merz, Michael & Wüthrich, Mario V., 2023. "Deep quantile and deep composite triplet regression," Insurance: Mathematics and Economics, Elsevier, vol. 109(C), pages 94-112.
    17. Ruodu Wang & Ričardas Zitikis, 2021. "An Axiomatic Foundation for the Expected Shortfall," Management Science, INFORMS, vol. 67(3), pages 1413-1429, March.
    18. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    19. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    20. Tobias Fissler & Jana Hlavinov'a & Birgit Rudloff, 2019. "Elicitability and Identifiability of Systemic Risk Measures," Papers 1907.01306, arXiv.org, revised Oct 2019.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:307:y:2023:i:3:p:1408-1423. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.