Large-scale text similarity computing with spark

Abstract

Text understanding is a hot research in Natural Language Processing and Information Retrieval. In recent years, it has received wide attention and research. In the era of big data, Understanding text in large-scale datasets is a challenge. Although the earliest systems designed for these workloads, such as MapReduce, gave users a powerful, but low-level, procedural programming interface. So, MapReduce doesn’t compose well for lager text applications. Recently, Spark, an in-memory cluster-computing platform, has been proposed. It has emerged as a popular framework for large-scale data processing and analytics. It provides a general-purpose efficient cluster computing engine and simpler for the end users. In this work, we consider using Vector Space Model (VSM) and TF-IDF weighting schema and feature hashing feature extraction techniques in order to solve the problem of large-scale text data similarity computing by Spark. As a result, Experimental results that using Spark in order to solve document similarity computation problems as soon as quickly by 20Newsgroups. In additions, It is more benefit from document classification and clustering of machine learning tasks.