Modeling agents working on ETL processes

Abstract

All data warehousing systems architects know that is quite difficult to build a “perfect” ETL system without having primarily a model well founded on real operational requisites. Thus ETL modeling is crucial to the success of populating a data warehouse, independently from their natural complexity. Modeling will also reduce implementation costs once it allows for revealing eventual operational errors originated by misinterpreted requisites. In this demo we demonstrate the viability of modeling an ETL system, using a standard workflow language and an ETL pattern approach implemented based on a cooperative community of opportunistic agents. The demo shows how we modeled, configured and executed an ETL system prototype in YAWL, and how we used agents to implement a self-organizing general purpose ETL system.