Ranking answers by hierarchical topic models

Abstract

Topic models are hierarchical probabilistic models for the statistical analysis of document collections. It assumes that each document comprises a mixture of latent topics and each topic can be represented by a distribution over vocabulary. Dimensionality for a large corpus of unstructured documents can be reduced by modeling with these exchangeable topics. In previous work, we designed a multi-pipe structure for question answering (QA) systems by nesting keyword search, classical Natural Language Processing (NLP) techniques and prototype detections. In this research, we use those technologies to select a set of sentences as candidate answers. We then use topic models to rank these candidate answers by calculating the semantic distances between these sentences and the given query. In our experiments, we found that the new model of using topic models improves the answer ranking so that the better answers can returned for the given query. © 2009 Springer Berlin Heidelberg.