Metadata handling for Big Data projects

Abstract

Metadata is information about information. In business, industry or long living scientific experiments metadata grow and evolve with the project lifecycle. It leads to changes in the structure of the metadata, and with time it becomes complex, sophisticated and fluid, so that even simple lookup request appears to be complicated enough to require special tools. Another issue is that metadata can be produced and stored in different ways - paper or digital documents and tables, or databases, or something very specific - depending on the initial capabilities and requirements to its utilization. Due to this, to have a holistic view of the project one often has to perform so called multi source requests, aggregating information from a number of different sources. This kind of requests is not easy to implement, and can hardly be used for online services due to the significant execution time. This paper describes a possible solution by suggesting a method of metadata integration organization and providing an example of its application to information infrastructure of a HEP experiment.