A multiple-criteria framework for evaluation of decision support systems

In the literature, decision support systems (DSSs) have typically been evaluated on only a single criterion such as the outcome from decision making. However, it is clear that DSSs simultaneously have a critical impact on the process-oriented aspects of decision making, suggesting that a combination of both outcome and process criteria are highly relevant for DSS evaluation. Indeed, process characteristics are particularly crucial for web-based and real-time DSSs because of their ability to deliver timely, current information through features such as just-in-time information, real-time processing, on-line transaction processing, connectivity and globally up-to-date information. In this underexplored area, we propose a framework to evaluate DSSs that combines outcome- and process-oriented evaluation measures. The approach is demonstrated in the context of a real-time threat criticality detection DSS. Investigations are conducted using a multicriteria decision-making method called the Analytic Hierarchy Process (AHP) as well as a newly developed stochastic enhancement of AHP. We find that the real-time DSS offered a significant improvement in terms of process-related characteristics. However, it did not offer a statistically significant improvement in terms of outcome-related characteristics. The importance of simultaneously addressing both sets of considerations is discussed.