A Comprehensive Spark‐Based Layer for Converting Relational Databases to NoSQL

Abstract

Currently, the continuous massive growth in the size, variety, and velocity of data is de-fined as big data. Relational databases have a limited ability to work with big data. Consequently, not only structured query language (NoSQL) databases were utilized to handle big data because NoSQL represents data in diverse models and uses a variety of query languages, unlike traditional relational databases. Therefore, using NoSQL has become essential, and many studies have at-tempted to propose different layers to convert relational databases to NoSQL; however, most of them targeted only one or two models of NoSQL, and evaluated their layers on a single node, not in a distributed environment. This study proposes a Spark‐based layer for mapping relational databases to NoSQL models, focusing on the document, column, and key–value databases of NoSQL models. The proposed Spark‐based layer comprises of two parts. The first part is concerned with converting relational databases to document, column, and key–value databases, and encompasses two phases: a metadata analyzer of relational databases and Spark‐based transformation and mi-gration. The second part focuses on executing a structured query language (SQL) on the NoSQL. The suggested layer was applied and compared with Unity, as it has similar components and fea-tures and supports sub‐queries and join operations in a single‐node environment. The experi-mental results show that the proposed layer outperformed Unity in terms of the query execution time by a factor of three. In addition, the proposed layer was applied to multi‐node clusters using different scenarios, and the results show that the integration between the Spark cluster and NoSQL databases on multi‐node clusters provided better performance in reading and writing while in-creasing the dataset size than using a single node.