A semantic overlapping clustering algorithm for analyzing short-texts

Abstract

The rise in volumes of digitized short-texts like tweets or customer complaints and opinions about products and services pose new challenges to the established methods of text analytics both due to the sparseness of text and noise. In this paper we present a new semantic clustering algorithm, which first discovers frequently occurring semantic concepts within a repository, and then clusters the documents around these concepts based on concept distribution within them. The method produces overlapping clusters which generates far more accurate view of content embedded within real-life communication texts. We have compared the clustering results with LSH based clustering and show that the proposed method produces fewer overall clusters with more semantic coherence within a cluster.