Hierarchical Summarization of Text Documents Using Topic Modeling and Formal Concept Analysis

Abstract

Availability of large collection of text documents triggers the need for large-scale text summarization. Identification of topics and organization of documents are important for analysis and exploration of textual data. This study is a part of the growing body of research on large-scale text summarization. It is an experimental approach and it differs from earlier works in the sense that it generates topic hierarchy which simultaneously provides a hierarchical structure to the document corpus. The documents are labeled with topics using latent Dirichlet allocation (LDA) and are automatically organized in a lattice structure using formal concept analysis (FCA). This groups the semantically related documents together. The lattice is further converted to a tree for better visualization and easy exploration of data. To signify the effectiveness of the approach, we have carried out the experiment and evaluation on 20 Newsgroup Dataset. Results depict that the presented method is considerably successful in forming topic hierarchy.