Character based pattern mining for neology detection

Abstract

Detecting neologisms is essential in real-time natural language processing applications. Not only can it enable to follow the lexical evolution of languages, but it is also essential for updating linguistic resources and parsers. In this paper, neology detection is considered as a classification task where a system has to assess whether a given lexical item is an actual neologism or not. We propose a combination of an unsupervised data mining technique and a supervised machine learning approach. It is inspired by current researches in stylometry and on tokenlevel and character-level patterns. We train and evaluate our system on a manually designed reference dataset in French and Russian. We show that this approach is able to outperform stateof- the-art neology detection systems. Furthermore, character-level patterns exhibit good properties for multilingual extensions of the system.