From Single Architectural Design to a Reference Conceptual Meta-Model: An Intelligent Data Lake for New Data Insights

Abstract

Recently, a new concept has appeared in the world of new technologies dealing with Big Data. This concept is called Data Lake (DL) and it is becoming the most suitable way to administer and put up Big Data new generation systems. Today's Big Data storage systems suffer from many problems related to data structure, accessibility and data quality. In order to resolve these issues, DL systems offer referential without schema for unprocessed data with a common access interface. Consequently, storing data in a DL without any metadata governance will only generate a “data swamp“. This paper describes a new architecture implementation for DL systems with optimal management of metadata. This process treats data from heterogeneous data sources and with a combination of Data Warehouse (DW) for better management of structured data. The proposal system discovers, extracts and classifies structural metadata from various data sources using ontologies. A new generation solution that makes available to all users sources of large amounts of information. This new approach offers companies an answer to data management problems and information availability.