Discovering concepts from word Co-occurrences with a relational model

Abstract

Clustering word co-occurrences has been studied to discover clusters as latent concepts. Previous work has applied the semantic aggregate model (SAM), and reports that discovered clusters seem semantically significant. The SAM assumes a co-occurrence arises from one latent concept. This assumption seems moderately natural. However, to analyze latent concepts more deeply, the assumption may be too restrictive. We propose to make clusters for each part of speech from co-occurrence data. For example, we make adjective clusters and noun clusters from adjective-noun co-occurrences while the SAM builds clusters of "co-occurrences." The proposed approach allows us to analyze adjectives and nouns independently. To take this approach, we propose a frequency-based infinite relational model (FIRM) for word co-occurrences. The FIRM is a stochastic block model that takes into account the frequency of observations although traditional stochastic blockmodels ignore it. The FIRM also utilizes the Dirichlet process so that the number of clusters is inferred. We derive a variational inference algorithm for the model to apply to a large dataset. Experimental results show that the FIRM is more helpful to analyze adjectives and nouns independently, and the FIRM clusters capture the SAM clusters better than a stochastic blockmodel.