System training metrics and measures: A key operational effectiveness imperative

An assessment framework to make explicit the “cause and effect” relationship between design decisions and their impact on system operations, maintenance, and support is essential to influence new and upgrade program development from the longer‐term life‐cycle perspective. This becomes even more urgent with increasingly greater utilization of commercial‐off‐the‐shelf (COTS) elements within information and knowledge intensive systems in the commercial (IT, Telecommunication, Banking, Finance) and aerospace domains. These architectures are often characterized by an evolving physical baseline (technology refreshment) driven by obsolescence and end‐of‐life risk considerations. The first objective of this paper is to present the concept of System Operational Effectiveness (SOE). System Operational Effectiveness serves as a generic framework for a wholistic system assessment by balancing factors pertaining to system performance, availability, process efficiency, and cost. Then, given the significance of system training costs, the results of an industry survey on system training metrics and methods are presented. This survey was conducted to help understand training metrics currently utilized within industry with a particular focus on information and knowledge intensive systems. A subsequent objective is to delineate architectural attributes that can be used to assess architectural goodness with respect to training requirements and cost. This is an ongoing research initiative and initial results from this initiative are also presented. © 2003 Wiley Periodicals, Inc. Syst Eng 6: 238–248, 2003