Native language identification by human judges

Abstract

This paper reports the results of two studies involving native language identification (NLI) by human judges (see Jarvis, 2012; Malmasi, Tetreault, & Dras, 2015; Odin, 1996). The first study includes six Finnish speakers who were asked to read a set of essays written in English and decide as quickly as possible whether each text was written by a Finnish or Swedish speaker. The second study includes the same six Finnish raters and an additional group of 10 Spanish-speaking raters. Both sets of raters were given 40 essays from the International Corpus of Learner English. The raters were asked to read each text carefully and indicate which texts they believed were written by native speakers of their own native languages. The results of both studies show that some of the raters were remarkably accurate in identifying the correct L1s of the writers, with accuracy rates above 80% for the top Finnish raters and over 90% for the top Spanish raters. A brief qualitative analysis of the data describes the types of features in the texts that allowed the raters to achieve such high accuracy rates. The paper concludes with a discussion of the theoretical, methodological, and pedagogical implications of these findings.