On the effect of stopword removal for SMS-based FAQ retrieval

Abstract

This paper investigates the effects of stopword removal in different stages of a system for SMS-based FAQ retrieval. Experiments are performed on the FIRE 2011 monolingual English data. The FAQ system comprises several stages, including normalization and correction of SMS, retrieval of FAQs potentially containing answers using the BM25 retrieval model, and detection of out-of-domain queries based on a k nearest-neighbor classifier. Both retrieval and OOD detection are tested with different stopword lists. Results indicate that i) retrieval performance is highest when stopwords are not removed and decreases when longer stopword lists are employed, ii) OOD detection accuracy decreases when trained on features collected during retrieval using no stopwords, iii) a combination of retrieval using no stopwords and OOD detection trained using the SMART stopwords yields the best results: 75.1% in-domain queries are answered correctly and 85.6% OOD queries are detected correctly. © 2012 Springer-Verlag.