Robust extraction of named entity including unfamiliar word

Abstract

This paper proposes a novel method to extract named entities including unfamiliar words which do not occur or occur few times in a training corpus using a large unannotated corpus. The proposed method consists of two steps. The first step is to assign the most similar and familiar word to each unfamiliar word based on their context vectors calculated from a large unannotated corpus. After that, traditional machine learning approaches are employed as the second step. The experiments of extracting Japanese named entities fromIREX corpus and NHK corpus show the effectiveness of the proposed method. © 2008 Association for Computational Linguistics.