ISM@FIRE-2011 Bengali Monolingual task: A frequency-based stemmer

Abstract

This paper describes the work that we did at Indian School of Mines, Dhanbad towards adhoc Bengali monolingual retrieval task for FIRE 2011. During official submissions, we prepared three TD runs using TERRIER search retrieval system without query expansion. When we used YASS stemmer we received substantially improved retrieval performance. Post-submission, we also developed a statistical stemmer based on frequent pattern mining using apriori-like algorithm taken from market basket data analysis. Initial results that we received for our stemmer showed noticeable retrieval performance gain over no-stem runs. Although this performance-gain is lower than that of YASS, we believe that it is promising enough to fine-tune the stemmer towards better results.