Bigger versus Similar: Selecting a background corpus for first story detection based on distributional similarity

1Citations
Citations of this article
60Readers
Mendeley users who have this article in their library.

Abstract

The current state of the art for First Story Detection (FSD) are nearest neighbour-based models with traditional term vector representations; however, one challenge faced by FSD models is that the document representation is usually defined by the vocabulary and term frequency from a background corpus. Consequently, the ideal background corpus should arguably be both large-scale to ensure adequate term coverage, and similar to the target domain in terms of the language distribution. However, given these two factors cannot always be mutually satisfied, in this paper we examine whether the distributional similarity of common terms is more important than the scale of common terms for FSD. As a basis for our analysis we propose a set of metrics to quantitatively measure the scale of common terms and the distributional similarity between corpora. Using these metrics we rank different background corpora relative to a target corpus. We also apply models based on different background corpora to the FSD task. Our results show that term distributional similarity is more predictive of good FSD performance than the scale of common terms; and, thus we demonstrate that a smaller recent domain-related corpus will be more suitable than a very large-scale general corpus for FSD.

Cite

CITATION STYLE

APA

Wang, F., Ross, R. J., & Kelleher, J. D. (2019). Bigger versus Similar: Selecting a background corpus for first story detection based on distributional similarity. In International Conference Recent Advances in Natural Language Processing, RANLP (Vol. 2019-September, pp. 1312–1320). Incoma Ltd. https://doi.org/10.26615/978-954-452-056-4_150

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free