A new event detection model based on term reweighting

Abstract

New event detection (NED) is aimed at detecting from one or multiple streams of news stories the one being reported on a new event (i.e. not reported previously). Preliminary experiments show that terms of different types (e.g. Noun and Verb) have different effects for different classes of stories in determining whether or not two stories are on the same topic. Unfortunately, conventional approaches usually ignore the fact. This paper proposes a NED model utilizing two approaches to addressing the problem based on term reweighting. In the first approach, the paper proposes to employ statistics on training data to learn the model for each class of stories, and in the second, the paper proposes to adjust term weights dynamically based on previous story clusters. Experimental results on two linguistic data consortium (LDC) data sets: TDT2 and TDT3 show that both the proposed approaches can effectively improve the performance of NED task, compared to the baseline method and existing methods.