Quality Assessment and Evaluation Criteria in Supervised Learning

Evaluating the performance of a learning algorithm is one of the basic tasks in machine learning and data science. In this chapter, we review commonly used performance measures and discuss their properties. We show that different measures focus on different aspects of the algorithm. Therefore, a learning algorithm is typically evaluated with respect to several criteria. We introduce conceptual tools and provide important guidelines for quality assessment of fully trained algorithms. We focus our attention to classification problems, as we draw connections to basic concepts in statistics, engineering, and other disciplines. In addition, we discuss regression problems, as we study popular residual-based measures. Finally, we suggest that evaluation criteria shall also be considered during the design of the algorithm. In this sense, the desired criteria determine the objective function, prior to the training of the algorithm. These design considerations are discussed, and several approaches are introduced to the problem.