Evaluating topic modeling interpretability using topic labeled gold-standard sets

Abstract

The paucity of rigorous evaluation measures undermines topic modeling results’ validity and trustworthiness. Accordingly, we propose a method that researchers can use to select models when they assess topics’ human interpretability. We show how they can evaluate different topic models using gold-standard sets that humans label. Our approach ensures that the topics extracted algorithmically from an entire corpus concur with the themes humans would have identified in the same documents. By doing so, we combine human coding’s advantages for topic interpretability with algorithmic topic Modeling’s analytical efficiency and scalability. We demonstrate that one can rigorously identify optimal model parametrizations for maximum interpretability and to rigorously justify model selection. We also contribute three open access gold-standard sets in the hospitality context and make them available so other researchers can use them to benchmark their models or validate their results. Finally, we showcase a methodology for designing and developing gold-standard sets for validating topic models, which researchers interested in developing gold-standard sets in domains and contexts appropriate for their research can use.