Intrinsic evaluations of word embeddings: What can we do better?

Abstract

This paper presents an analysis of existing methods for the intrinsic evaluation of word embeddings. We show that the main methodological premise of such evaluations is "interpretability" of word embeddings: A "good" embedding produces results that make sense in terms of traditional linguistic categories. This approach is not only of limited practical use, but also fails to do justice to the strengths of distributional meaning representations. We argue for a shift from abstract ratings of word embedding "quality" to exploration of their strengths and weaknesses.