IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v36y2016i4p653-665.html
   My bibliography  Save this article

Validation in the Absence of Observed Events

Author

Listed:
  • John Lathrop
  • Barry Ezell

Abstract

This article addresses the problem of validating models in the absence of observed events, in the area of weapons of mass destruction terrorism risk assessment. We address that problem with a broadened definition of “validation,” based on stepping “up” a level to considering the reason why decisionmakers seek validation, and from that basis redefine validation as testing how well the model can advise decisionmakers in terrorism risk management decisions. We develop that into two conditions: validation must be based on cues available in the observable world; and it must focus on what can be done to affect that observable world, i.e., risk management. That leads to two foci: (1) the real‐world risk generating process, and (2) best use of available data. Based on our experience with nine WMD terrorism risk assessment models, we then describe three best use of available data pitfalls: SME confidence bias, lack of SME cross‐referencing, and problematic initiation rates. Those two foci and three pitfalls provide a basis from which we define validation in this context in terms of four tests—Does the model: … capture initiation? … capture the sequence of events by which attack scenarios unfold? … consider unanticipated scenarios? … consider alternative causal chains? Finally, we corroborate our approach against three validation tests from the DOD literature: Is the model a correct representation of the process to be simulated? To what degree are the model results comparable to the real world? Over what range of inputs are the model results useful?

Suggested Citation

  • John Lathrop & Barry Ezell, 2016. "Validation in the Absence of Observed Events," Risk Analysis, John Wiley & Sons, vol. 36(4), pages 653-665, April.
  • Handle: RePEc:wly:riskan:v:36:y:2016:i:4:p:653-665
    DOI: 10.1111/risa.12442
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/risa.12442
    Download Restriction: no

    File URL: https://libkey.io/10.1111/risa.12442?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gregory S. Parnell & Christopher M. Smith & Frederick I. Moxley, 2010. "Intelligent Adversary Risk Analysis: A Bioterrorism Risk Management Model," Risk Analysis, John Wiley & Sons, vol. 30(1), pages 32-48, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Michael Greenberg & Anthony Cox & Vicki Bier & Jim Lambert & Karen Lowrie & Warner North & Michael Siegrist & Felicia Wu, 2020. "Risk Analysis: Celebrating the Accomplishments and Embracing Ongoing Challenges," Risk Analysis, John Wiley & Sons, vol. 40(S1), pages 2113-2127, November.
    2. Zachary A. Collier & James H. Lambert, 2019. "Principles and methods of model validation for model risk reduction," Environment Systems and Decisions, Springer, vol. 39(2), pages 146-153, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jason R. W. Merrick & Laura A. McLay, 2010. "Is Screening Cargo Containers for Smuggled Nuclear Threats Worthwhile?," Decision Analysis, INFORMS, vol. 7(2), pages 155-171, June.
    2. Aakil M. Caunhye & Xiaofeng Nie, 2018. "A Stochastic Programming Model for Casualty Response Planning During Catastrophic Health Events," Transportation Science, INFORMS, vol. 52(2), pages 437-453, March.
    3. Michael R. Greenberg & Karen Lowrie & Henry Mayer & Tayfur Altiok, 2011. "Risk‐Based Decision Support Tools: Protecting Rail‐Centered Transit Corridors from Cascading Effects," Risk Analysis, John Wiley & Sons, vol. 31(12), pages 1849-1858, December.
    4. Jason Merrick & Gregory S. Parnell, 2011. "A Comparative Analysis of PRA and Intelligent Adversary Methods for Counterterrorism Risk Management," Risk Analysis, John Wiley & Sons, vol. 31(9), pages 1488-1510, September.
    5. Andrew G. Huff & James S. Hodges & Shaun P. Kennedy & Amy Kircher, 2015. "Evaluation of the Food and Agriculture Sector Criticality Assessment Tool (FASCAT) and the Collected Data," Risk Analysis, John Wiley & Sons, vol. 35(8), pages 1448-1467, August.
    6. David Rios Insua & David Banks & Jesus Rios, 2016. "Modeling Opponents in Adversarial Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 36(4), pages 742-755, April.
    7. G. Quijano, Eduardo & Ríos Insua, David & Cano, Javier, 2018. "Critical networked infrastructure protection from adversaries," Reliability Engineering and System Safety, Elsevier, vol. 179(C), pages 27-36.
    8. Vicki Bier, 2020. "The Role of Decision Analysis in Risk Analysis: A Retrospective," Risk Analysis, John Wiley & Sons, vol. 40(S1), pages 2207-2217, November.
    9. Edouard Kujawski, 2016. "A Probabilistic Game‐Theoretic Method to Assess Deterrence and Defense Benefits of Security Systems," Systems Engineering, John Wiley & Sons, vol. 19(6), pages 549-566, November.
    10. Juan Carlos Sevillano & David Rios Insua & Jesus Rios, 2012. "Adversarial Risk Analysis: The Somali Pirates Case," Decision Analysis, INFORMS, vol. 9(2), pages 86-95, June.
    11. Vineet M. Payyappalli & Jun Zhuang & Victor Richmond R. Jose, 2017. "Deterrence and Risk Preferences in Sequential Attacker–Defender Games with Continuous Efforts," Risk Analysis, John Wiley & Sons, vol. 37(11), pages 2229-2245, November.
    12. Dogucan Mazicioglu & Jason R. W. Merrick, 2018. "Behavioral Modeling of Adversaries with Multiple Objectives in Counterterrorism," Risk Analysis, John Wiley & Sons, vol. 38(5), pages 962-977, May.
    13. Gilberto Montibeller & L. Alberto Franco & Ashley Carreras, 2020. "A Risk Analysis Framework for Prioritizing and Managing Biosecurity Threats," Risk Analysis, John Wiley & Sons, vol. 40(11), pages 2462-2477, November.
    14. Jason R. W. Merrick & Philip Leclerc, 2016. "Modeling Adversaries in Counterterrorism Decisions Using Prospect Theory," Risk Analysis, John Wiley & Sons, vol. 36(4), pages 681-693, April.
    15. Xiaojun Shan & Jun Zhuang, 2013. "Cost of Equity in Homeland Security Resource Allocation in the Face of a Strategic Attacker," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1083-1099, June.
    16. Sumitra Sri Bhashyam & Gilberto Montibeller, 2012. "Modeling State-Dependent Priorities of Malicious Agents," Decision Analysis, INFORMS, vol. 9(2), pages 172-185, June.
    17. Jinshu Cui & Heather Rosoff & Richard S. John, 2016. "Cumulative Response to Sequences of Terror Attacks Varying in Frequency and Trajectory," Risk Analysis, John Wiley & Sons, vol. 36(12), pages 2272-2284, December.
    18. Sumitra Sri Bhashyam & Gilberto Montibeller, 2016. "In the Opponent's Shoes: Increasing the Behavioral Validity of Attackers’ Judgments in Counterterrorism Models," Risk Analysis, John Wiley & Sons, vol. 36(4), pages 666-680, April.
    19. Jesus Rios & David Rios Insua, 2012. "Adversarial Risk Analysis for Counterterrorism Modeling," Risk Analysis, John Wiley & Sons, vol. 32(5), pages 894-915, May.
    20. Tianyang Wang & James S. Dyer & John C. Butler, 2016. "Modeling Correlated Discrete Uncertainties in Event Trees with Copulas," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 396-410, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:36:y:2016:i:4:p:653-665. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.