IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v96y2011i9p1242-1256.html
   My bibliography  Save this article

The Method of Manufactured Universes for validating uncertainty quantification methods

Author

Listed:
  • Stripling, H.F.
  • Adams, M.L.
  • McClarren, R.G.
  • Mallick, B.K.

Abstract

The Method of Manufactured Universes is presented as a validation framework for uncertainty quantification (UQ) methodologies and as a tool for exploring the effects of statistical and modeling assumptions embedded in these methods. The framework calls for a manufactured reality from which “experimental†data are created (possibly with experimental error), an imperfect model (with uncertain inputs) from which simulation results are created (possibly with numerical error), the application of a system for quantifying uncertainties in model predictions, and an assessment of how accurately those uncertainties are quantified. The application presented in this paper manufactures a particle-transport “universe†, models it using diffusion theory with uncertain material parameters, and applies both Gaussian process and Bayesian MARS algorithms to make quantitative predictions about new “experiments†within the manufactured reality. The results of this preliminary study indicate that, even in a simple problem, the improper application of a specific UQ method or unrealized effects of a modeling assumption may produce inaccurate predictions. We conclude that the validation framework presented in this paper is a powerful and flexible tool for the investigation and understanding of UQ methodologies.

Suggested Citation

  • Stripling, H.F. & Adams, M.L. & McClarren, R.G. & Mallick, B.K., 2011. "The Method of Manufactured Universes for validating uncertainty quantification methods," Reliability Engineering and System Safety, Elsevier, vol. 96(9), pages 1242-1256.
    • Handle: RePEc:eee:reensy:v:96:y:2011:i:9:p:1242-1256
    DOI: 10.1016/j.ress.2010.11.012
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832011000755
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2010.11.012?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Higdon, Dave & Gattiker, James & Williams, Brian & Rightley, Maria, 2008. "Computer Model Calibration Using High-Dimensional Output," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 570-583, June.
    2. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Drignei, Dorin, 2011. "A general statistical model for computer experiments with time series output," Reliability Engineering and System Safety, Elsevier, vol. 96(4), pages 460-467.
    2. Hwang, Youngdeok & Kim, Hang J. & Chang, Won & Yeo, Kyongmin & Kim, Yongku, 2019. "Bayesian pollution source identification via an inverse physics model," Computational Statistics & Data Analysis, Elsevier, vol. 134(C), pages 76-92.
    3. Ioannis Andrianakis & Ian R Vernon & Nicky McCreesh & Trevelyan J McKinley & Jeremy E Oakley & Rebecca N Nsubuga & Michael Goldstein & Richard G White, 2015. "Bayesian History Matching of Complex Infectious Disease Models Using Emulation: A Tutorial and a Case Study on HIV in Uganda," PLOS Computational Biology, Public Library of Science, vol. 11(1), pages 1-18, January.
    4. Samantha M. Roth & Ben Seiyon Lee & Sanjib Sharma & Iman Hosseini‐Shakib & Klaus Keller & Murali Haran, 2023. "Flood hazard model calibration using multiresolution model output," Environmetrics, John Wiley & Sons, Ltd., vol. 34(2), March.
    5. Perrin, G., 2020. "Adaptive calibration of a computer code with time-series output," Reliability Engineering and System Safety, Elsevier, vol. 196(C).
    6. White, Staci A. & Herbei, Radu, 2015. "A Monte Carlo approach to quantifying model error in Bayesian parameter estimation," Computational Statistics & Data Analysis, Elsevier, vol. 83(C), pages 168-181.
    7. Mevin Hooten & Christopher Wikle & Michael Schwob, 2020. "Statistical Implementations of Agent‐Based Demographic Models," International Statistical Review, International Statistical Institute, vol. 88(2), pages 441-461, August.
    8. Leatherman, Erin R. & Dean, Angela M. & Santner, Thomas J., 2017. "Designing combined physical and computer experiments to maximize prediction accuracy," Computational Statistics & Data Analysis, Elsevier, vol. 113(C), pages 346-362.
    9. Wu, Xu & Kozlowski, Tomasz & Meidani, Hadi, 2018. "Kriging-based inverse uncertainty quantification of nuclear fuel performance code BISON fission gas release model using time series measurement data," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 422-436.
    10. Nott, David J. & Marshall, Lucy & Fielding, Mark & Liong, Shie-Yui, 2014. "Mixtures of experts for understanding model discrepancy in dynamic computer models," Computational Statistics & Data Analysis, Elsevier, vol. 71(C), pages 491-505.
    11. Paulo, Rui & García-Donato, Gonzalo & Palomo, Jesús, 2012. "Calibration of computer models with multivariate output," Computational Statistics & Data Analysis, Elsevier, vol. 56(12), pages 3959-3974.
    12. Jackson Samuel E. & Vernon Ian & Liu Junli & Lindsey Keith, 2020. "Understanding hormonal crosstalk in Arabidopsis root development via emulation and history matching," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 19(2), pages 1-33, April.
    13. Guillaume Perrin & Christian Soize, 2020. "Adaptive method for indirect identification of the statistical properties of random fields in a Bayesian framework," Computational Statistics, Springer, vol. 35(1), pages 111-133, March.
    14. Matthew Plumlee, 2014. "Fast Prediction of Deterministic Functions Using Sparse Grid Experimental Designs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1581-1591, December.
    15. SungKu Kang & Ran Jin & Xinwei Deng & Ron S. Kenett, 2023. "Challenges of modeling and analysis in cybermanufacturing: a review from a machine learning and computation perspective," Journal of Intelligent Manufacturing, Springer, vol. 34(2), pages 415-428, February.
    16. Manfren, Massimiliano & Aste, Niccolò & Moshksar, Reza, 2013. "Calibration and uncertainty analysis for computer models – A meta-model based approach for integrated building energy simulation," Applied Energy, Elsevier, vol. 103(C), pages 627-641.
    17. Daniel W. Gladish & Daniel E. Pagendam & Luk J. M. Peeters & Petra M. Kuhnert & Jai Vaze, 2018. "Emulation Engines: Choice and Quantification of Uncertainty for Complex Hydrological Models," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 23(1), pages 39-62, March.
    18. Chen, Yewen & Chang, Xiaohui & Luo, Fangzhi & Huang, Hui, 2023. "Additive dynamic models for correcting numerical model outputs," Computational Statistics & Data Analysis, Elsevier, vol. 187(C).
    19. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    20. Bruno Sansó & Chris Forest, 2009. "Statistical calibration of climate system properties," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 58(4), pages 485-503, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:96:y:2011:i:9:p:1242-1256. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.