Autoscale: Automatic ETL scale process

Abstract

In this paper we investigate the problem of providing automatic scalability and data freshness to data warehouses, when at the same time dealing with high-rate data efficiently. In general, data freshness is not guaranteed in those contexts, since data loading, transformation and integration are heavy tasks that are performed only periodically, instead of row by row. Many current data warehouse deployments are designed to be deployed and work in a single server. However, for many applications problems related with data volume processing times, data rates and requirements for fresh and fast responses, require new solutions to be found. The solution is to use/build parallel architectures and mechanisms to speed-up data integration and to handle fresh data efficiently. Desirably, users developing data warehouses need to concentrate solely on the conceptual and logic design (e.g. business driven requirements, logical warehouse schemas, workload and ETL process), while physical details, including mechanisms for scalability, freshness and integration of high-rate data, should be left to automated tools. We propose a universal data warehouse parallelization solution, that is, an approach that enables the automatic scalability and freshness of any data warehouse and ETL process. Our results show that the proposed system can handle scalablity to provide the desired processing speed.