Trend graphs: Visualizing the evolution of concept relationships in large document collections

Abstract

The proliferation of digitally available textual data necessitates automatic tools for analyzing large textual collections. Thus, in analogy to data mining for structured databases, text mining is defined for textual collections. A central tool in text mining is the analysis of concept relationship, which discovers connections between different concepts, as reflected in the corpus. Most previous work on text mining in general, and concept relationship in particular, viewed the entire corpus as one monolithic entity. However, large corpuses are often composed of documents with different characteristics. Most importantly, documents are often tagged with timestamps (e.g. news articles), and thus represent the state of the domain in different time periods. In this paper we introduce a new technique for analyzing and visualizing differences and similarities in the concept relationships, as they are reflected in different segments of the corpus. Focusing on the case of timestamped documents, we introduce Trend Graphs, which provide a graphical tool for analyzing and visualizing the dynamic changes in concept relationships over time. Trend Graphs thus provide a tool for tracking the evaluation of the corpus over time, highlighting trends and discontinuities.