A statistical approach for multilingual document clustering and topic extraction from clusters

Abstract

This paper describes a statistics-based methodology for document unsupervised clustering and cluster topics extraction. For this purpose, multiword lexical units (MWUs) of any length are automatically extracted from corpora using the LiPXtractor - a language independent statistics-based tool. The MWUs are taken as base features to characterize documents. These features are transformed and a document similarity matrix is constructed. From this matrix, a reduced set of features is selected using an approach based on Principal Component Analisys. Then, using the Model Based Clustering Analisys software, it is possible to obtain the best number of clusters. Precision and Recall for document-cluster assignment range above 90 %. Most important MWUs are extracted from each cluster and taken as document cluster topics. Results on new document classification will just be mentioned.