IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v195y2020ics0951832019302571.html
   My bibliography  Save this article

A general model validation and testing tool

Author

Listed:
  • Vanslette, Kevin
  • Tohme, Tony
  • Youcef-Toumi, Kamal

Abstract

We construct and propose the “Bayesian Validation Metric†(BVM) as a general model validation and testing tool. We find the BVM to be capable of representing all of the standard validation metrics (square error, reliability, probability of agreement, frequentist, area, probability density comparison, statistical hypothesis testing, and Bayesian model testing) as special cases and find that it can be used to improve, generalize, or further quantify their uncertainties. Thus, the BVM allows us to assess the similarities and differences between existing validation metrics in a new light.

Suggested Citation

  • Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    • Handle: RePEc:eee:reensy:v:195:y:2020:i:c:s0951832019302571
    DOI: 10.1016/j.ress.2019.106684
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832019302571
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2019.106684?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhao, Lufeng & Lu, Zhenzhou & Yun, Wanying & Wang, Wenjin, 2017. "Validation metric based on Mahalanobis distance for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 80-89.
    2. Wu, Danqing & Lu, Zhenzhou & Wang, Yanping & Cheng, Lei, 2015. "Model validation and calibration based on component functions of model output," Reliability Engineering and System Safety, Elsevier, vol. 140(C), pages 59-70.
    3. Rebba, Ramesh & Mahadevan, Sankaran, 2008. "Computational methods for model reliability assessment," Reliability Engineering and System Safety, Elsevier, vol. 93(8), pages 1197-1207.
    4. Li, Chenzhao & Mahadevan, Sankaran, 2016. "Role of calibration, validation, and relevance in multi-level uncertainty integration," Reliability Engineering and System Safety, Elsevier, vol. 148(C), pages 32-43.
    5. Sankararaman, Shankar & Mahadevan, Sankaran, 2011. "Model validation under epistemic uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 96(9), pages 1232-1241.
    6. Sankararaman, Shankar & Mahadevan, Sankaran, 2015. "Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems," Reliability Engineering and System Safety, Elsevier, vol. 138(C), pages 194-209.
    7. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    8. Ling, You & Mahadevan, Sankaran, 2013. "Quantitative model validation techniques: New insights," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 217-231.
    9. Mullins, Joshua & Ling, You & Mahadevan, Sankaran & Sun, Lin & Strachan, Alejandro, 2016. "Separation of aleatory and epistemic uncertainty in probabilistic model validation," Reliability Engineering and System Safety, Elsevier, vol. 147(C), pages 49-59.
    10. Wang, Chong & Matthies, Hermann G., 2019. "Novel model calibration method via non-probabilistic interval characterization and Bayesian theory," Reliability Engineering and System Safety, Elsevier, vol. 183(C), pages 84-92.
    11. John Geweke, 2007. "Bayesian Model Comparison and Validation," American Economic Review, American Economic Association, vol. 97(2), pages 60-64, May.
    12. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    13. Trucano, T.G. & Swiler, L.P. & Igusa, T. & Oberkampf, W.L. & Pilch, M., 2006. "Calibration, validation, and sensitivity analysis: What's what," Reliability Engineering and System Safety, Elsevier, vol. 91(10), pages 1331-1357.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tohme, Tony & Vanslette, Kevin & Youcef-Toumi, Kamal, 2023. "Reliable neural networks for regression uncertainty estimation," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    2. Khastgir, Siddartha & Brewerton, Simon & Thomas, John & Jennings, Paul, 2021. "Systems Approach to Creating Test Scenarios for Automated Driving Systems," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    3. Pietrantuono, Roberto & Popov, Peter & Russo, Stefano, 2020. "Reliability assessment of service-based software under operational profile uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    4. Tohme, Tony & Vanslette, Kevin & Youcef-Toumi, Kamal, 2020. "A generalized Bayesian approach to model calibration," Reliability Engineering and System Safety, Elsevier, vol. 204(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tohme, Tony & Vanslette, Kevin & Youcef-Toumi, Kamal, 2020. "A generalized Bayesian approach to model calibration," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    2. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    3. Wang, Chong & Matthies, Hermann G., 2019. "Novel model calibration method via non-probabilistic interval characterization and Bayesian theory," Reliability Engineering and System Safety, Elsevier, vol. 183(C), pages 84-92.
    4. Sankararaman, Shankar & Mahadevan, Sankaran, 2015. "Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems," Reliability Engineering and System Safety, Elsevier, vol. 138(C), pages 194-209.
    5. Ao, Dan & Hu, Zhen & Mahadevan, Sankaran, 2017. "Design of validation experiments for life prediction models," Reliability Engineering and System Safety, Elsevier, vol. 165(C), pages 22-33.
    6. Jung, Yongsu & Jo, Hwisang & Choo, Jeonghwan & Lee, Ikjin, 2022. "Statistical model calibration and design optimization under aleatory and epistemic uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    7. Jung, Yongsu & Lee, Ikjin, 2021. "Optimal design of experiments for optimization-based model calibration using Fisher information matrix," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    8. McKeand, Austin M. & Gorguluarslan, Recep M. & Choi, Seung-Kyum, 2021. "Stochastic analysis and validation under aleatory and epistemic uncertainties," Reliability Engineering and System Safety, Elsevier, vol. 205(C).
    9. Perrin, G., 2020. "Adaptive calibration of a computer code with time-series output," Reliability Engineering and System Safety, Elsevier, vol. 196(C).
    10. Park, Inseok & Grandhi, Ramana V., 2014. "A Bayesian statistical method for quantifying model form uncertainty and two model combination methods," Reliability Engineering and System Safety, Elsevier, vol. 129(C), pages 46-56.
    11. Ray, Douglas & Ramirez-Marquez, Jose, 2020. "A framework for probabilistic model-based engineering and data synthesis," Reliability Engineering and System Safety, Elsevier, vol. 193(C).
    12. Drignei, Dorin, 2011. "A general statistical model for computer experiments with time series output," Reliability Engineering and System Safety, Elsevier, vol. 96(4), pages 460-467.
    13. Yuan, Jun & Ng, Szu Hui, 2013. "A sequential approach for stochastic computer model calibration and prediction," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 273-286.
    14. Li, Luyi & Lu, Zhenzhou, 2018. "A new method for model validation with multivariate output," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 579-592.
    15. Kwag, Shinyoung & Gupta, Abhinav & Dinh, Nam, 2018. "Probabilistic risk assessment based model validation method using Bayesian network," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 380-393.
    16. Wu, Danqing & Lu, Zhenzhou & Wang, Yanping & Cheng, Lei, 2015. "Model validation and calibration based on component functions of model output," Reliability Engineering and System Safety, Elsevier, vol. 140(C), pages 59-70.
    17. Yoo, Yeongmin & Jung, Ui-Jin & Han, Yong Ha & Lee, Jongsoo, 2021. "Data Augmentation-Based Prediction of System Level Performance under Model and Parameter Uncertainties: Role of Designable Generative Adversarial Networks (DGAN)," Reliability Engineering and System Safety, Elsevier, vol. 206(C).
    18. Maupin, Kathryn A. & Swiler, Laura P., 2020. "Model discrepancy calibration across experimental settings," Reliability Engineering and System Safety, Elsevier, vol. 200(C).
    19. Ling, You & Mahadevan, Sankaran, 2013. "Quantitative model validation techniques: New insights," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 217-231.
    20. Zhao, Lufeng & Lu, Zhenzhou & Yun, Wanying & Wang, Wenjin, 2017. "Validation metric based on Mahalanobis distance for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 80-89.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:195:y:2020:i:c:s0951832019302571. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.