Analyzing Data from Multiple Raters

Abstract

At the end of the last chapter, we examined the relationship between an expert's ratings of a set of high school student's essays and several easily quantifiable attributes measured from these essays. The particular essay grades that we examined happened to be the grades from the first of five experts who graded the essays. However, with more than one expert grader, an obvious question becomes "How would our analysis change if we used another expert's ratings, or if we somehow combined the grades from all experts?" Collecting ordinal data from multiple raters is a bit like having more than one wristwatch. With one watch, you know the time-or at least you think you do. But with two watches, you are never sure. The same phenomenon exists with multiple raters. In the previous chapter, we assumed that the "true grade" of each essay was known, and then we analyzed the essays to assess the relationship between these grades and various grammatical attributes. Unfortunately, when we examine ratings from several raters, it generally happens that the classifications assigned to individuals by different raters are not consistent. We must therefore decide how to combine the information gathered from different raters. Numerous approaches have been proposed for analyzing ordinal data collected from multiple raters. Often, emphasis in such analyses focuses on modeling the agreement between raters. Among the more commonly used indices of multirater agreement in social sciences and medicine is the κ statistic (Cohen 1960). Assuming that all judges employ the same number of rating categories, the κ statistic can be estimated by constructing a contingency table in which each judge is treated as