A chance-corrected measure of inter-annotator agreement for syntax

12Citations
Citations of this article
105Readers
Mendeley users who have this article in their library.

Abstract

Following the works of Carletta (1996) and Artstein and Poesio (2008), there is an increasing consensus within the field that in order to properly gauge the reliability of an annotation effort, chance-corrected measures of inter-annotator agreement should be used. With this in mind, it is striking that virtually all evaluations of syntactic annotation efforts use uncorrected parser evaluation metrics such as bracket F1 (for phrase structure) and accuracy scores (for dependencies). In this work we present a chance-corrected metric based on Krippendorff's , adapted to the structure of syntactic annotations and applicable both to phrase structure and dependency annotation without any modifications. To evaluate our metric we first present a number of synthetic experiments to better control the sources of noise and gauge the metric's responses, before finally contrasting the behaviour of our chance-corrected metric with that of uncorrected parser evaluation metrics on real corpora. © 2014 Association for Computational Linguistics.

References Powered by Scopus

A Coefficient of Agreement for Nominal Scales

31146Citations
N/AReaders
Get full text

Reliability of content analysis: The case of nominal scale coding

1374Citations
N/AReaders
Get full text

Inter-coder agreement for computational linguistics

1171Citations
N/AReaders
Get full text

Cited by Powered by Scopus

Phrase structure annotation and parsing for learner English

20Citations
N/AReaders
Get full text

Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks

13Citations
N/AReaders
Get full text

Exploring ensemble dependency parsing to reduce manual annotation workload

1Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Skjærholt, A. (2014). A chance-corrected measure of inter-annotator agreement for syntax. In 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014 - Proceedings of the Conference (Vol. 1, pp. 934–944). Association for Computational Linguistics (ACL). https://doi.org/10.3115/v1/p14-1088

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 42

67%

Researcher 11

17%

Professor / Associate Prof. 7

11%

Lecturer / Post doc 3

5%

Readers' Discipline

Tooltip

Computer Science 49

75%

Linguistics 12

18%

Neuroscience 2

3%

Engineering 2

3%

Save time finding and organizing research with Mendeley

Sign up for free