Distributed parallelization of semantic web java applications by means of the message-passing interface

Abstract

In the recent years, performance has become a key point for a number of Java applications. For some of them, such as from the Semantic Web domain, where the size and the scale of the analyzed data is of a big challenge for a conventional computer, use of the High Performance Computing (HPC) systems is a major factor in achieving the required scalability and performance demands. Parallelization is a key mechanism that leverages HPC for such applications. However, the high development effort for a scalable parallel application has been a major drawback towards the efficient application of HPC to the applications designed for a serial execution only. The Message-Passing Interface (MPI) is a well-known programming standard for large-scale parallel applications development. However, MPI has found its most wide use in the applications written in C and Fortran. We show how MPI can be beneficially applied for the parallelization of the Java applications as well. We describe a parallel implementation of a Random Indexing application that performs similarity search in the large text corpora on the web, which allowed us to improve the performance by up to 33 times on the already 16 nodes of a testbed HPC system. © Springer-Verlag Berlin Heidelberg 2012.