Hierarchical dirichlet process topic modeling for large number of answer types classification in open domain question answering

Abstract

We propose a new method that uses the Hierarchical Dirichlet Process (HDP) to classify a large number of answer types for a question posed using natural language. We used the HDP model to build a classifier that assigns test questions to certain clusters, then computes similarity among the questions within the same cluster. Our answer-type classifier finds the n-best similar training questions to the test questions and classifies the test question’s answer type as the majority of the n-best training question’s answer type. The proposed method achieved similar accuracy and lower sensitivity to the presence of a large number of answer types than existing methods that use classification algorithms with same features. Also, we can guarantee that appropriate answer type can be among the ranked answer types with high recall.