Topic detection and document similarity on financial news

Abstract

Traders often rely on financial news to come up with predictions for stock price changes. Dealing with vast amount of news data makes it essential to use an automated methodology to identify the relevant news items for a given criteria. In this study we use Latent Dirichlet Allocation (LDA) to model the correlation of news items with stock price time series data. LDA model is trained with news items from a time window in the past and then the trained model is used to measure the similarity between the current news items and the news items used for training. Calculated similarity measure can be used as a predictor for switching points in the future. We tested our methodology using a collection of about 1,700,000 financial news items published between 2015-01-01 and 2015-12-31, and compared the results with various standard classification techniques. Our results indicate that use of LDA instead of standard classification techniques makes it possible to achieve the same level of performance by using a much smaller feature space.