Improving Performance of English-Hindi CLIR System using Linguistic Tools and Techniques

Abstract

World Wide Web is growing rapidly and the content on Web of languages other than English is also increasing rapidly compared to English. Hindi is most widely spoken language in India. In past few years Hindi content has also increased rapidly on the Web. To ensure complete information exchange, in the era of globalization the information retrieval systems need to be multilingual or cross lingual. We have designed and developed an English-Hindi Cross Language Information Retrieval (CLIR) System using Dictionary based query translation method. Our previous experiments [5] showed reasonable 64.80% performance of the monolingual retrieval with this system using the TREC style test collection created especially for this research. This paper describes results of the English-Hindi CLIR experiments using some specialized query formulation strategies like stopword removal, stemming of query terms, transliteration of out of vocabulary words etc. The results demonstrated that the performance gradually improved when we applied NLP tools and techniques in short queries. Performance was dropped down to some extent when using query expansion and structuring as well using long queries to obtained cross-language results. The best performance result we obtained from these experiments was 82.91% compared to the monolingual retrieval.