Consistency of Ranking Aggregation Techniques

For a specific ranking aggregation problem, the application of different aggregation techniques may lead to different results. For the aggregation to be effective, the resulting collective judgment should reflect the relevant input data, i.e., experts’ rankings and importance hierarchy. This chapter describes some quantitative tools to check the degree of consistency between the collective judgment and the input data, in a practical and intuitive way. The first family of tools—called p indicators—is applicable to a wide variety of practical contexts, such as problems in which experts’ rankings are complete/incomplete and the importance hierarchy is expressed in different forms. Another tool is the indicator W k m + 1 $$ {W}_k^{\left(m+1\right)} $$ , which is derived from the Kendall’s concordance coefficient. Several practical examples accompany the description.