Topic Modelling with NMF vs. Expert Topic Annotation: The Case Study of Russian Fiction

Abstract

The paper presents an experiment aimed at comparison of results of topic modelling via non-negative matrix factorization (NMF) with that of manual topic annotation performed by an expert. The experiment was conducted on the annotated corpus of Russian short stories of the initial three decades of the 20th century, which contains 310 stories with a total of 1000000 tokens written by 300 Russian writers. The annotation scheme used in topic annotation includes 89 topics, further this list was reduced down to 30 generalized ones, the most frequent of which turned out to be the following: death, relationships, love, social groups, social processes, family, money, human sins, nature, religion, and war. Then, the corpus divided into three consecutive time periods was subjected to NMF topic modelling which provided a model including 24 topics. The results of both topic annotations were compared and described. The paper discusses the main findings of the study and the difficulties of fiction topic modelling which should be taken into account. For example, experimental results showed that topic modelling via NMF should be primarily recommended for the revealing of topics referring to general background of literary texts (e.g., war, love, nature, family) rather than for detecting topics related with some critical events or relations between characters (e.g., death or relations). The comparison of human and automatic topic annotation seems an important step for the improvement of artificial technologies techniques related with NLP.