An approach to inter-annotator agreement evaluation for the named entities annotation task at opencorpora

Abstract

In this paper we introduce an approach towards inter-annotator agreement (IAA) evaluation for tasks with an unrestricted set of possible choices. IAA is an important tool because it provides direct feedback to both annotators and annotation guidelines developers. There are several approaches towards IAA evaluation for tasks with a closed set of alternatives like morphological annotation; adaptation of these to tasks with an unrestricted set of possible choices is not fully applicable. An aposteriori evaluation approach is introduced in this paper in order to evaluate current inter-annotator agreement for crowdsourcing on the stage before moderation. The proposed method is based on Venn diagrams naturally representing opinions of multiple annotators selecting an arbitrary number of word sequences. Four categories of agreement are pointed out, and the contribution of each is considered and visualized along with proportions of their investment. One total criterion is selected to evaluate the agreement. The approbation results are discussed for evaluating data of named entities annotation task in the OpenCorpora project in Russian. The recommendations are given on the approach applicability for evaluating IAA in unmoderated named entities annotation data with open set of alternatives.