Diverse demands estimation and ranking based on user behaviors

Abstract

In the big data era, users can get massive information from the Internet, but the value density is very low. In order to help users find the information they need more quickly, this paper presents the mechanism of diverse demands estimation and ranking based on user behaviors. Firstly, a definition of classification system for users query intent is proposed. Secondly, in order to mine the documents on the websites of specific classification, LDA model is used to cluster and annotate the websites. To speed up the inference process of LDA, we take advantage of MPI and OpenMP hybrid parallelism techniques to reduce both internode and intra-node communication cost. Lastly, according to the historical behaviors of users and the search engine return results, we rank the classifications on Map-Reduce platform and present the top-ranking ones to users