LDA based event extraction: Detecting influenza epidemics using microblog

Abstract

As a major public health concern, influenza epidemics causes tens of millions respiratory illnesses worldwide each year. With the development of social network, interaction platform like microblog, is generating massive data providing us a faster and more accurate way to predict the trends in the spread of influenza, which can help us reduce the impact cause by the influenza. The problem of influenza epidemics prediction through Chinese microblog cannot be easily addressed by applying existing approaches and methods, some of which have been used for English documents. Besides, different from traditional text, the microblog is big in volume, update velocity, noise and small in the individual text volume, which cause that traditional deeper semantic analysis method like SVM is inefficient and easy to be over-fitting. To address this problem, we present a deeper semantic analysis to Chinese microblog using a LDA based event extraction framework. Our experiment using 332,886 microblogs from south and north China showed that our method achieved more detailed information extraction about the flu and an earlier flu prediction than the Chinese official ILI data.