Using a time granularity table for gradual granular data aggregation

Abstract

The majority of today's systems increasingly require sophisticated data management as they need to store and to query large amounts of data for analysis and reporting purposes. In order to keep more "detailed" data available for longer periods, "old" data has to be reduced gradually to save space and improve query performance, especially on resource-constrained systems with limited storage and query processing capabilities. A number of data reduction solutions have been developed, however an effective solution particularly based on gradual data reduction is missing. This paper presents an effective solution for data reduction based on gradual granular data aggregation. With the gradual granular data aggregation mechanism, older data can be made coarse-grained while keeping the newest data fine-grained. For instance, when data is 3 months old aggregate to 1 minute level from 1 second level, when data is 6 months old aggregate to 2 minutes level from 1 minute level and so on. The proposed solution introduces a time granularity based data structure, namely a relational time granularity table that enables long term storage of old data by maintaining it at different levels of granularity and effective query processing due to a reduction in data volume. In addition, the paper describes the implementation strategy derived from a farming case study using standard technologies. © 2010 Springer-Verlag.