Approximate set similarity join using many-core processors

Abstract

Set similarity join is a database operation used to find out all similar pairs of sets from two collections over sets. Due to the high versatility, it has been applied in many applications in various domains, including text processing, image processing, etc. One of its drawbacks is the high computational costs, especially when the size of the given collections is large. In this paper, we propose a scheme for efficient set similarity join on Intel Xeon Phi, one of the latest many core processor. In order to make best use of high computational power of Intel Xeon Phi, we employ following approaches: (1) we compress each record by b-bit MinHash to fit them in MCDRAM which is a high bandwidth on-chip memory; and (2) we apply some optimizations such as utilization of 512-bit SIMD instructions and loop unrolling. Experimental results show that our proposed method outperforms CPU implementation.