Using Rule and Goal Based Agents to Create Metadata Profiles

Abstract

Good quality metadata can be a contributing factor when it comes to large scale data integration. In order to minimize data fetch and access request in data lakes, metadata can present adequate solutions that require minimal data access provided metadata exists. Metadata discovery can help us understand how data semantics operate, intrinsic and extrinsic data relationships as well as features that guide query processing, data management, and data integration. Metadata is mostly generated using manual annotation or is discovered through data profiling. What we are looking to explore as a part of our research is to understand available metadata and create profiles that can serve as ‘menu card’ for the other datasets in the data lake. In this paper, we present a technique for generating metadata profiles using goal based and rule-based agents. To this end, we apply simple rules and guide agents with actionable goals to attain an automatic categorization of a metadata file. Our technique was evaluated experimentally, the results show that applied techniques allow comparing multiple metadata profiles in order to compute similarity and difference measures.