A real-time distributed index based on topic for microblogging system

Abstract

With the development of internet technology and widely used in mobile devices, the microblogging systems such as Twitter and Sina Weibo in China have become the most important platform for people to retrieve information and communicate with each other. The real-time search became a big challenge for microblogging systems because of the volume of data and users. Existing approaches build all microblogs in an index which will increase the cost of index update and query. The search results could not satisfy users’ timely and high quality requirements. In this paper, we propose a new real-time distributed index based on topic (RDIBT), which can build index for each topic. Those topical indices will be distributed to many sites, so it can improve the concurrently of queries. Extensive experiments demonstrate the effectiveness and efficiency of RDIBT on the real dataset.