Opening machine translation black box for cross-language information retrieval

Abstract

State-of-the-art Statistical Machine Translation (SMT) systems are widely used for query translation in Cross-Language Information Retrieval (CLIR), but usually as a black box. A strong limitation is that only one-best translation is retained. It is known that CLIR can benefit much from using multiple translation alternatives which produces a query expansion effect. It is then desirable to extend the one-best translation output to a richer translation including more translation alternatives. In fact, translation alternatives are available in SMT before the final best output is selected. A natural way is to open the black box of SMT to access the internal search graph in order to select more translation alternatives. In this paper, we consider the translation alternatives included in the N-best translation outputs. Using our approach for CLIR, we report up to 40% improvement in Mean Average Precision over baseline query translation using an SMT system as a black box on TREC-9 English-Chinese CLIR task, and 10% on NTCIR-5 English-Chinese task. This study demonstrates the usefulness of opening the black box of SMT to produce more adequate query translations for CLIR. © Springer-Verlag 2012.