Continuous-space language processing: Beyond word embeddings

Abstract

Spoken and written language processing has seen a dramatic shift in recent years to increased use of continuous-space representations of language via neural networks and other distributional methods. In particular, word embeddings are used in many applications. This paper looks at the advantages of the continuous-space approach and limitations of word embeddings, reviewing recent work that attempts to model more of the structure in language. In addition, we discuss how current models characterize the exceptions in language and opportunities for advances by integrating traditional and continuous approaches.