Building topic/trend detection system based on slow intelligence

Abstract

It becomes an interesting research topic to detect trend in the Internet era, where millions of data are posted online everyday. As social media, for example, blogs, forums, and micro-blogs, are prevailing, many offline events are discussed online. The discussion data, which reflects what people are interested in, is useful for detecting trend. This research proposes a design of online topic/trend detection system with the advantages of Slow Intelligence. Unlike traditional Topic Detection and Tracking (TDT) tasks, which source data from offline news articles, the proposed system attempts to collect and analyze huge amount of up-to-date data from many heterogeneous websites on Internet. The Internet data complicates the system in two aspects: 1) it needs careful resource allocation to collect huge amount of up-to-date data based on limited computing resources; 2) it needs mechanisms to automatically or semi-automatically adapt data processing algorithms to handle varieties of data. This research adopts Slow Intelligence, which provides a framework for systems with insufficient computing resources to gradually adapt to environments, to handle these complexities.