A Statistical Approach to Assessment of Agreement Involving Multiple Raters

Study of agreement between two or more comparable sets of measurements taken on each member of a population is needed in many areas. There is an impressive literature on the topic covering both the qualitative and quantitative features of study variables. In order to ascertain the extent of agreement between the quantitative ‘target’ variable X and the quantitative ‘test’ variable Y, coverage probability (CP) was introduced by Lin et al. (2002). CP is defined as the probability of (X, Y) falling into a strip along the direction of X = Y of a judiciously specified width 2d but symmetric with respect to X and Y. For a given d, the higher CP value is, the better agreement it indicates. This article dwells on the concept of agreement involving three or more comparable quantitative measurements taken on each member of a population. We contemplate on a situation wherein we have available a reference gold standard against which several competitors are to be compared with respect to their performance as judged by the CP. We also address the problem of judging the performance of the competitors in the absence of any prescribed gold standard.