IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v40y2020i12p2639-2660.html
   My bibliography  Save this article

Fighting the Curse of Sparsity: Probabilistic Sensitivity Measures From Cumulative Distribution Functions

Author

Listed:
  • Elmar Plischke
  • Emanuele Borgonovo

Abstract

Quantitative models support investigators in several risk analysis applications. The calculation of sensitivity measures is an integral part of this analysis. However, it becomes a computationally challenging task, especially when the number of model inputs is large and the model output is spread over orders of magnitude. We introduce and test a new method for the estimation of global sensitivity measures. The new method relies on the intuition of exploiting the empirical cumulative distribution function of the simulator output. This choice allows the estimators of global sensitivity measures to be based on numbers between 0 and 1, thus fighting the curse of sparsity. For density‐based sensitivity measures, we devise an approach based on moving averages that bypasses kernel‐density estimation. We compare the new method to approaches for calculating popular risk analysis global sensitivity measures as well as to approaches for computing dependence measures gathering increasing interest in the machine learning and statistics literature (the Hilbert–Schmidt independence criterion and distance covariance). The comparison involves also the number of operations needed to obtain the estimates, an aspect often neglected in global sensitivity studies. We let the estimators undergo several tests, first with the wing‐weight test case, then with a computationally challenging code with up to k=30,000 inputs, and finally with the traditional Level E benchmark code.

Suggested Citation

  • Elmar Plischke & Emanuele Borgonovo, 2020. "Fighting the Curse of Sparsity: Probabilistic Sensitivity Measures From Cumulative Distribution Functions," Risk Analysis, John Wiley & Sons, vol. 40(12), pages 2639-2660, December.
    • Handle: RePEc:wly:riskan:v:40:y:2020:i:12:p:2639-2660
    DOI: 10.1111/risa.13571
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/risa.13571
    Download Restriction: no
    File URL: https://libkey.io/10.1111/risa.13571?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wei, Pengfei & Lu, Zhenzhou & Song, Jingwen, 2015. "Variable importance analysis: A comprehensive review," Reliability Engineering and System Safety, Elsevier, vol. 142(C), pages 399-432.
    2. Storlie, Curtis B. & Swiler, Laura P. & Helton, Jon C. & Sallaberry, Cedric J., 2009. "Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models," Reliability Engineering and System Safety, Elsevier, vol. 94(11), pages 1735-1763.
    3. E. Borgonovo & S. Tarantola & E. Plischke & M. D. Morris, 2014. "Transformations and invariance in the sensitivity analysis of computer experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(5), pages 925-947, November.
    4. Emanuele Borgonovo & William Castaings & Stefano Tarantola, 2011. "Moment Independent Importance Measures: New Results and Analytical Test Cases," Risk Analysis, John Wiley & Sons, vol. 31(3), pages 404-428, March.
    5. Bowman, A. W. & Azzalini, A., 2003. "Computational aspects of nonparametric smoothing with illustrations from the sm library," Computational Statistics & Data Analysis, Elsevier, vol. 42(4), pages 545-560, April.
    6. Mark Strong & Jeremy E. Oakley, 2013. "An Efficient Method for Computing Single-Parameter Partial Expected Value of Perfect Information," Medical Decision Making, , vol. 33(6), pages 755-766, August.
    7. Ronald L. Iman & Mark E. Johnson & Charles C. Watson, 2005. "Uncertainty Analysis for Computer Model Projections of Hurricane Losses," Risk Analysis, John Wiley & Sons, vol. 25(5), pages 1299-1312, October.
    8. Emanuele Borgonovo & Gordon B. Hazen & Elmar Plischke, 2016. "A Common Rationale for Global Sensitivity Measures and Their Estimation," Risk Analysis, John Wiley & Sons, vol. 36(10), pages 1871-1895, October.
    9. H. Christopher Frey & Sumeet R. Patil, 2002. "Identification and Review of Sensitivity Analysis Methods," Risk Analysis, John Wiley & Sons, vol. 22(3), pages 553-578, June.
    10. Tarantola, S. & Gatelli, D. & Mara, T.A., 2006. "Random balance designs for the estimation of first order global sensitivity indices," Reliability Engineering and System Safety, Elsevier, vol. 91(6), pages 717-727.
    11. Andrea Saltelli & Stefano Tarantola & Karen Chad, 1998. "Presenting Results from Model Based Studies to Decision‐Makers: Can Sensitivity Analysis Be a Defogging Agent?," Risk Analysis, John Wiley & Sons, vol. 18(6), pages 799-803, December.
    12. Storlie, Curtis B. & Helton, Jon C., 2008. "Multiple predictor smoothing methods for sensitivity analysis: Description of techniques," Reliability Engineering and System Safety, Elsevier, vol. 93(1), pages 28-54.
    13. James C. Felli & Gordon B. Hazen, 1998. "Sensitivity Analysis and the Expected Value of Perfect Information," Medical Decision Making, , vol. 18(1), pages 95-109, January.
    14. Andreas Tsanakas & Pietro Millossovich, 2016. "Sensitivity Analysis Using Risk Measures," Risk Analysis, John Wiley & Sons, vol. 36(1), pages 30-48, January.
    15. Helton, Jon C. & Sallaberry, Cedric J., 2009. "Computational implementation of sampling-based approaches to the calculation of expected dose in performance assessments for the proposed high-level radioactive waste repository at Yucca Mountain, Nev," Reliability Engineering and System Safety, Elsevier, vol. 94(3), pages 699-721.
    16. E. E. Koks & M. Bočkarjova & H. de Moel & J. C. J. H. Aerts, 2015. "Integrated Direct and Indirect Flood Risk Modeling: Development and Sensitivity Analysis," Risk Analysis, John Wiley & Sons, vol. 35(5), pages 882-900, May.
    17. Sudret, Bruno, 2008. "Global sensitivity analysis using polynomial chaos expansions," Reliability Engineering and System Safety, Elsevier, vol. 93(7), pages 964-979.
    18. Ted G. Eschenbach, 1992. "Spiderplots versus Tornado Diagrams for Sensitivity Analysis," Interfaces, INFORMS, vol. 22(6), pages 40-46, December.
    19. Kleijnen, J.P.C. & Bettonvil, B.W.M., 1997. "Searching for important factors in simulation models with many factors : Sequential bifurcation," Other publications TiSEM be826993-22f9-4cb3-89df-3, Tilburg University, School of Economics and Management.
    20. Saltelli A. & Tarantola S., 2002. "On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 702-709, September.
    21. Kleijnen, J.P.C., 2017. "Design and Analysis of simulation experiments : Tutorial," Other publications TiSEM c7ad6b68-dcd6-4485-9ee2-0, Tilburg University, School of Economics and Management.
    22. Emanuele Borgonovo, 2006. "Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches," Risk Analysis, John Wiley & Sons, vol. 26(5), pages 1349-1361, October.
    23. J. C. Helton & F. J. Davis, 2002. "Illustration of Sampling‐Based Methods for Uncertainty and Sensitivity Analysis," Risk Analysis, John Wiley & Sons, vol. 22(3), pages 591-622, June.
    24. Ronald L. Iman & Stephen C. Hora, 1990. "A Robust Measure of Uncertainty Importance for Use in Fault Tree System Analysis," Risk Analysis, John Wiley & Sons, vol. 10(3), pages 401-406, September.
    25. Jon C. Helton, 1994. "Treatment of Uncertainty in Performance Assessments for Complex Systems," Risk Analysis, John Wiley & Sons, vol. 14(4), pages 483-511, August.
    26. Bettonvil, Bert & Kleijnen, Jack P. C., 1997. "Searching for important factors in simulation models with many factors: Sequential bifurcation," European Journal of Operational Research, Elsevier, vol. 96(1), pages 180-194, January.
    27. Mark Strong & Jeremy E. Oakley & Jim Chilcott, 2012. "Managing structural uncertainty in health economic decision models: a discrepancy approach," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 61(1), pages 25-45, January.
    28. R. A. Riedmann & B. Gasic & D. Vernez, 2015. "Sensitivity Analysis, Dominant Factors, and Robustness of the ECETOC TRA v3, Stoffenmanager 4.5, and ART 1.5 Occupational Exposure Models," Risk Analysis, John Wiley & Sons, vol. 35(2), pages 211-225, February.
    29. Sumeet R. Patil & H. Christopher Frey, 2004. "Comparison of Sensitivity Analysis Methods Based on Applications to a Food Safety Risk Assessment Model," Risk Analysis, John Wiley & Sons, vol. 24(3), pages 573-585, June.
    30. Gabor J. Szekely & Maria L. Rizzo, 2005. "Hierarchical Clustering via Joint Between-Within Distances: Extending Ward's Minimum Variance Method," Journal of Classification, Springer;The Classification Society, vol. 22(2), pages 151-183, September.
    31. Plischke, Elmar & Borgonovo, Emanuele & Smith, Curtis L., 2013. "Global sensitivity measures from given data," European Journal of Operational Research, Elsevier, vol. 226(3), pages 536-550.
    32. Helton, J.C. & Hansen, C.W. & Sallaberry, C.J., 2014. "Expected dose for the nominal scenario class in the 2008 performance assessment for the proposed high-level radioactive waste repository at Yucca Mountain, Nevada," Reliability Engineering and System Safety, Elsevier, vol. 122(C), pages 267-271.
    33. Andrea Saltelli, 2002. "Sensitivity Analysis for Importance Assessment," Risk Analysis, John Wiley & Sons, vol. 22(3), pages 579-590, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stefano Cucurachi & Carlos Felipe Blanco & Bernhard Steubing & Reinout Heijungs, 2022. "Implementation of uncertainty analysis and moment‐independent global sensitivity analysis for full‐scale life cycle assessment models," Journal of Industrial Ecology, Yale University, vol. 26(2), pages 374-391, April.
    2. Zdeněk Kala, 2022. "Quantification of Model Uncertainty Based on Variance and Entropy of Bernoulli Distribution," Mathematics, MDPI, vol. 10(21), pages 1-19, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. S. Cucurachi & E. Borgonovo & R. Heijungs, 2016. "A Protocol for the Global Sensitivity Analysis of Impact Assessment Models in Life Cycle Assessment," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 357-377, February.
    2. Emanuele Borgonovo & Gordon B. Hazen & Elmar Plischke, 2016. "A Common Rationale for Global Sensitivity Measures and Their Estimation," Risk Analysis, John Wiley & Sons, vol. 36(10), pages 1871-1895, October.
    3. Borgonovo, Emanuele & Plischke, Elmar, 2016. "Sensitivity analysis: A review of recent advances," European Journal of Operational Research, Elsevier, vol. 248(3), pages 869-887.
    4. Isadora Antoniano‐Villalobos & Emanuele Borgonovo & Sumeda Siriwardena, 2018. "Which Parameters Are Important? Differential Importance Under Uncertainty," Risk Analysis, John Wiley & Sons, vol. 38(11), pages 2459-2477, November.
    5. Tatsuya Sakurahara & Seyed Reihani & Ernie Kee & Zahra Mohaghegh, 2020. "Global importance measure methodology for integrated probabilistic risk assessment," Journal of Risk and Reliability, , vol. 234(2), pages 377-396, April.
    6. Barry Anderson & Emanuele Borgonovo & Marzio Galeotti & Roberto Roson, 2014. "Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help?," Risk Analysis, John Wiley & Sons, vol. 34(2), pages 271-293, February.
    7. Emanuele Borgonovo & William Castaings & Stefano Tarantola, 2011. "Moment Independent Importance Measures: New Results and Analytical Test Cases," Risk Analysis, John Wiley & Sons, vol. 31(3), pages 404-428, March.
    8. Mirko Ginocchi & Ferdinanda Ponci & Antonello Monti, 2021. "Sensitivity Analysis and Power Systems: Can We Bridge the Gap? A Review and a Guide to Getting Started," Energies, MDPI, vol. 14(24), pages 1-59, December.
    9. Emanuele Borgonovo, 2006. "Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches," Risk Analysis, John Wiley & Sons, vol. 26(5), pages 1349-1361, October.
    10. Wei, Pengfei & Lu, Zhenzhou & Song, Jingwen, 2015. "Variable importance analysis: A comprehensive review," Reliability Engineering and System Safety, Elsevier, vol. 142(C), pages 399-432.
    11. Sinan Xiao & Zhenzhou Lu & Pan Wang, 2018. "Multivariate Global Sensitivity Analysis Based on Distance Components Decomposition," Risk Analysis, John Wiley & Sons, vol. 38(12), pages 2703-2721, December.
    12. Plischke, Elmar & Borgonovo, Emanuele & Smith, Curtis L., 2013. "Global sensitivity measures from given data," European Journal of Operational Research, Elsevier, vol. 226(3), pages 536-550.
    13. Hu, Zhen & Mahadevan, Sankaran, 2019. "Probability models for data-Driven global sensitivity analysis," Reliability Engineering and System Safety, Elsevier, vol. 187(C), pages 40-57.
    14. Emanuele Borgonovo, 2008. "Sensitivity Analysis of Model Output with Input Constraints: A Generalized Rationale for Local Methods," Risk Analysis, John Wiley & Sons, vol. 28(3), pages 667-680, June.
    15. Plischke, Elmar & Borgonovo, Emanuele, 2019. "Copula theory and probabilistic sensitivity analysis: Is there a connection?," European Journal of Operational Research, Elsevier, vol. 277(3), pages 1046-1059.
    16. Xuefei Lu & Alessandro Rudi & Emanuele Borgonovo & Lorenzo Rosasco, 2020. "Faster Kriging: Facing High-Dimensional Simulators," Operations Research, INFORMS, vol. 68(1), pages 233-249, January.
    17. Pesenti, Silvana M. & Millossovich, Pietro & Tsanakas, Andreas, 2019. "Reverse sensitivity testing: What does it take to break the model?," European Journal of Operational Research, Elsevier, vol. 274(2), pages 654-670.
    18. Helton, Jon C. & Brooks, Dusty M. & Sallaberry, Cédric J., 2020. "Margins associated with loss of assured safety for systems with multiple weak links and strong links," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    19. Pengfei Wei & Zhenzhou Lu & Jingwen Song, 2014. "Moment‐Independent Sensitivity Analysis Using Copula," Risk Analysis, John Wiley & Sons, vol. 34(2), pages 210-222, February.
    20. Emanuele Borgonovo, 2008. "Epistemic Uncertainty in the Ranking and Categorization of Probabilistic Safety Assessment Model Elements: Issues and Findings," Risk Analysis, John Wiley & Sons, vol. 28(4), pages 983-1001, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:40:y:2020:i:12:p:2639-2660. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.