Defining and assessing risk analysis quality: insights from applications of the SRA risk analysis quality test

The Applied Risk Management Specialty Group of the Society for Risk Analysis (SRA) identified a need to define, characterize, and improve risk analysis quality, specifically its quality in supporting risk management. To address that need, they drew on prior research and experience to develop the Risk Analysis Quality Test, the RAQT, a list of 76 questions, each asking if a risk analysis satisfies an aspect of risk analysis quality. The RAQT is both a definition of risk analysis quality, and a “spotter” of shortfalls, providing a language with which to describe then address possible shortfalls. The 76 questions were compiled by a working group based on risk science knowledge and on shortfalls they had observed in practice. With this study, we demonstrate that, simply by defining an explicit process to characterize risk analysis quality, the RAQT can improve risk analysis quality on several levels. We describe applications of the RAQT at each of three levels: 1) to evaluate risk analysis reporting within a large project; 2) to critique and suggest improvements for describing risk; and 3) more strategically, as a basis for orienting an organizational culture around awareness and management of risk. Finally, we discuss the implications of our study. This paper contributes to the risk analysis body of knowledge and practice by demonstrating the critical role of risk analysis quality assessment to identify shortfalls in risk characterization and communication, and the role of organizational culture in shaping how effectively risk analyses can guide risk management.