Word particles applied to information retrieval

Abstract

Document retrieval systems conventionally use words as the basic unit of representation, a natural choice since words are primary carriers of semantic information. In this paper we propose the use of a different, phonetically defined unit of representation that we call "particles". Particles are phonetic sequences that do not possess meaning. Both documents and queries are converted from their standard wordbased form into sequences of particles. Indexing and retrieval is performed with particles. Experiments show that this scheme is capable of achieving retrieval performance that is comparable to that from words when the text in the documents and queries are clean, and can result in significantly improved retrieval when they are noisy. © Springer-Verlag Berlin Heidelberg 2009.