Psychological Bulletin

Abstract

A previously described coefficient of agreement for nominal scales, kappa, treats all disagreements equally. A generalization to weighted kappa (KW) is presented. The KW provides for the incorporation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k X k table of joint nominal scale assignments such that disagreements of varying gravity (or agreements of varying degree) are weighted accordingly. Although providing for partial credit, KW is fully chance corrected. Its sampling characteristics and procedures for hypothesis testing and setting confidence limits are given. Under certain conditions, KW equals productmoment r. Although developed originally as a measure of reliability, the use of unequal weights for symmetrical cells makes KW suitable as a measure of validity.