Clustering massive text data streams by semantic smoothing model

Abstract

Clustering text data streams is an important issue in data mining community and has a number of applications such as news group filtering, text crawling, document organization and topic detection and tracing etc. However, most methods are similarity-based approaches and use the TF*IDF scheme to represent the semantics of text data and often lead to poor clustering quality. In this paper, we firstly give an improved semantic smoothing model for text data stream environment. Then we use the improved semantic model to improve the clustering quality and present an online clustering algorithm for clustering massive text data streams. In our algorithm, a new cluster statistics structure, cluster profile, is presented in which the semantics of text data streams are captured. We also present the experimental results illustrating the effectiveness of our technique. © Springer-Verlag Berlin Heidelberg 2007.