Measures of Concordance for Assessing Agreement in Ratings and Rank Order Data

Abstract

Consider an array of ratings or rank orders provided by M judges or respondents on n subjects or products. We propose a new general measure of concordance (agreement) in this context. This measure when apphed to rank order data reduces to Kendall's (1948) measure of concordance. A new measure for rank order data that depends on the average Kendall's measure of rank correlation is also proposed and is also a special case of our general measure. This particular rank order measure can be considered as an alternative to the well-known Friedman's statistic for the two-way analysis of variance. The general measure is also compared with another measure proposed by Lin (1989). Relations to the intraclass correlation coefficient of the ratings data are pointed out. Some distributional results are also presented. The proposed concordance measures provide the basis for testing the agreement between two or more methods, instruments or respondents in biometric research or market research surveys.