Comparison on the Effectiveness of Different Statistical Similarity Measures

Abstract

Document retrieval is the process of matching of some sated user query against a set of free-text records (documents), its one major technique for organizing and managing information. This project was concerned with studying which of the different statistical measures in IR have the most effectiveness on document retrieval using a unified set of documents. The results show that the Cosine Similarity Measure is the best of other seven measures (Inner Product, Dice Coefficient, Jaccard Coefficient, Inclusion Similarity Coefficient, Overlap Coefficient Measure, Euclidean distance Measure and Manhattan Distance Measure (City Block Distance) for both languages, with precision on Arabic collection 38% and recall 53.2%. On English collection, the precision is 25% and recall 65%.