Evolving an algorithm to generate sparse inverted index using hadoop and pig

Abstract

Now a day’s users mostly prefer the keyword search method to access the data for the explosion of information. Inverted indexing efficiently plays a very important role for search operation over a large set of data. There are two problems exist in current keyword based searching technique. First, the large set of data is mostly unstructured and does not suite in the existing database systems. Second, the storage in inverted indexing is usually very large and compression techniques used so far is also not so efficient because they increase the processing time. To overcome these problems, Hadoop, which is a distributed framework for large dataset is needed where the required resources could be shared and accessed very easily. In our proposed work, we will join the list of consecutive document id in the inverted index into the intervals to save memory space. For this, we have developed the UDF (User Defined Function) for stemming and stop words for the sparse inverted index in pig latin. It can be observed in the results that our proposed method is efficient than existing techniques.