Nonsense!: Quality control via two-step reason selection for annotating local acceptability and related attributes in news editorials

Abstract

Annotation quality control is a critical aspect for building reliable corpora through linguistic annotation. In this study, we present a simple but powerful quality control method using two-step reason selection. We gathered sentential annotations of local acceptability and three related attributes through a crowdsourcing platform. For each attribute, the reason for the choice of the attribute value is selected in a two-step manner. The options given for reason selection were designed to facilitate the detection of a nonsensical reason selection. We assume that a reliable annotation may not contain a nonsensical reason selected for the choice of the attribute value, and an annotation that contains a nonsensical reason is less reliable than the one without such reason. Our method, based solely on this assumption, is found to retain the annotations with remarkable quality out of the entire annotations mixed with those of low quality.