On Metadata Support for Integrating Evolving Heterogeneous Data Sources

Abstract

With the emergence of big data technologies, the problem of structure evolution of integrated heterogeneous data sources has become extremely topical due to dynamic and diverse nature of big data. To solve the big data evolution problem, we propose an architecture that allows to store and process structured and unstructured data at different levels of detail, analyze them using OLAP capabilities and semi-automatically manage changes in requirements and data expansion. In this paper, we concentrate on the metadata essential for the operation of the proposed architecture. We propose a metadata model to describe schemata and supplementary properties of data sets extracted from sources and transformed to obtain integrated data for the analysis in a flexible way. Furthermore, the unique feature of the proposed model is that it allows to keep track of all changes that occur in the system.