IDEAS home Printed from https://ideas.repec.org/a/wly/syseng/v17y2014i2p237-248.html

Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method

Author

Listed:
  • Jaemyung Ahn
  • Olivier L. de Weck
  • Martin Steele

Abstract

This paper introduces a procedure to assess the credibility of models and simulations (M&S) as a group activity based on NASA’s new standard for M&S NASA‐STD‐7009. The Delphi method, which is characterized by iterative surveys with controlled feedback, was selected to implement the assessment. The proposed procedure is expected to address the issues in the M&S assessment related to a high level of required expertise and group decision making. An actual credibility assessment study using the proposed procedure on an M&S platform referred to as SpaceNet has been carried out by ten panel members through a two‐round Delphi. The study concluded that the overall credibility of SpaceNet version 1.3 was between the development level and production level. The variances of the assessments in the second‐round survey were significantly reduced compared with the first‐round results, which indicates the effectiveness of the proposed procedure.

Suggested Citation

  • Jaemyung Ahn & Olivier L. de Weck & Martin Steele, 2014. "Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method," Systems Engineering, John Wiley & Sons, vol. 17(2), pages 237-248, June.
    • Handle: RePEc:wly:syseng:v:17:y:2014:i:2:p:237-248
    DOI: 10.1002/sys.21266
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/sys.21266
    Download Restriction: no
    File URL: https://libkey.io/10.1002/sys.21266?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    2. Litsiou, Konstantia & Polychronakis, Yiannis & Karami, Azhdar & Nikolopoulos, Konstantinos, 2022. "Relative performance of judgmental methods for forecasting the success of megaprojects," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1185-1196.
    3. Christoph Markmann & Alexander Spickermann & Heiko A. von der Gracht & Alexander Brem, 2021. "Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior," Futures & Foresight Science, John Wiley & Sons, vol. 3(1), March.
    4. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    5. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    6. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    7. Christoph Diermann & Arnd Huchzermeier, 2017. "Case Article—Canyon Bicycles: Judgmental Demand Forecasting in Direct Sales," INFORMS Transactions on Education, INFORMS, vol. 17(2), pages 58-62, January.
    8. Bo Cowgill & Eric Zitzewitz, 2015. "Corporate Prediction Markets: Evidence from Google, Ford, and Firm X," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1309-1341.
    9. Jukrin Moon & Dongoo Lee & Taesik Lee & Jaemyung Ahn & Jindong Shin & Kyungho Yoon & Dongsik Choi, 2015. "Group Decision Procedure to Model the Dependency Structure of Complex Systems: Framework and Case Study for Critical Infrastructures," Systems Engineering, John Wiley & Sons, vol. 18(4), pages 323-338, July.
    10. Tommaso Ciarli & Alex Coad & Ismael Rafols, 2015. "Quantitative Analysis of Technology Futures: A review of Techniques, Uses and Characteristics," SPRU Working Paper Series 2015-23, SPRU - Science Policy Research Unit, University of Sussex Business School.
    11. Muhammad Ridhuan Tony Lim Abdullah & Saedah Siraj & Zulkipli Ghazali, 2021. "An ISM Approach for Managing Critical Stakeholder Issues Regarding Carbon Capture and Storage (CCS) Deployment in Developing Asian Countries," Sustainability, MDPI, vol. 13(12), pages 1-23, June.
    12. repec:eee:intfor:v:27:y:2011:i:1:p:1-13 is not listed on IDEAS
    13. Apreda, Riccardo & Bonaccorsi, Andrea & dell'Orletta, Felice & Fantoni, Gualtiero, 2019. "Expert forecast and realized outcomes in technology foresight," Technological Forecasting and Social Change, Elsevier, vol. 141(C), pages 277-288.
    14. Cary Deck & David Porter, 2013. "Prediction Markets In The Laboratory," Journal of Economic Surveys, Wiley Blackwell, vol. 27(3), pages 589-603, July.
    15. Griffiths, Frances & Cave, Jonathan & Boardman, Felicity & Ren, Justin & Pawlikowska, Teresa & Ball, Robin & Clarke, Aileen & Cohen, Alan, 2012. "Social networks – The future for health care delivery," Social Science & Medicine, Elsevier, vol. 75(12), pages 2233-2241.
    16. Sung, Ming-Chien & McDonald, David C.J. & Johnson, Johnnie E.V. & Tai, Chung-Ching & Cheah, Eng-Tuck, 2019. "Improving prediction market forecasts by detecting and correcting possible over-reaction to price movements," European Journal of Operational Research, Elsevier, vol. 272(1), pages 389-405.
    17. Rengarajan, Srinath & Moser, Roger & Narayanamurthy, Gopalakrishnan, 2021. "Strategy tools in dynamic environments – An expert-panel study," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    18. Bolger, Fergus & Rowe, Gene & Belton, Ian & Crawford, Megan M & Hamlin, Iain & Sissons, Aileen & Taylor Browne Lūka, Courtney & Vasilichi, Alexandrina & Wright, George, 2020. "The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques," OSF Preprints 4ufzg, Center for Open Science.
    19. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.
    20. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    21. Tiberius, Victor & Gojowy, Robin & Dabić, Marina, 2022. "Forecasting the future of robo advisory: A three-stage Delphi study on economic, technological, and societal implications," Technological Forecasting and Social Change, Elsevier, vol. 182(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:syseng:v:17:y:2014:i:2:p:237-248. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)1520-6858 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.