COMPARING HDFS – GREENPLUM DATA LOADING OPTIONS

Abstract

In the last five years, many companies around the world have been successfully implemented Apache Hadoop as a main Data Lake storage for all data presented in the organization. At the same time, the adoption of other Open-Source technologies has been also increasing over years, such as classical MPP-based systems for Analytical workloads. Thus, the issue of efficient and fast data integration of Apache Hadoop and other organizational data storage systems becomes highly important for enterprises, where business and decision makers require the minimum delay of heterogeneous data exchange between Hadoop and other storages. In this paper, we compare different options for loading data from Apache Hadoop, representing the Data Lake of organization, into Open-Source MPP Greenplum database with the role of classical data warehouse for analytical workloads, and choose the best one. Also, we identify potential risks of using different data loading methods.