Data Integration Revitalized: From Data Warehouse Through Data Lake to Data Mesh

Abstract

For years, data integration (DI) architectures evolved from those supporting virtual integration, through physical integration, to those supporting both virtual and physical integration. Regardless of its type, all of the developed DI architectures include an integration layer. This layer is implemented by a sophisticated software, which runs the so-called DI processes. The integration layer is responsible for ingesting data from various sources (typically heterogeneous and distributed) and for homogenizing data into formats suitable for future processing and analysis. Nowadays, in all business domains, large volumes of highly heterogeneous data are produced, e.g., medical systems, smart cities, smart agriculture, which require further advancements in the data integration technologies. In this keynote talk paper, I present my personal opinion on still-to-be developed data integration techniques - potential research directions, namely: (1) more flexible DI, (2) quality assurance in complex multi-modal systems, (3) execution optimization of DI processes.