Flexible selection of wavelet coefficients based on the estimation error of predefined queries

Abstract

In this paper, we introduce a data stream reduction method using lossy wavelets compression. The lossy compression means that compressed data carry as much information about the original data stream as possible while the original data size remarkably reduced. We think that wavelets technique should be an efficient method for such lossy compression. Especially we consider storing a plenty of past data stream into stable storage (flash memory or micro HDD) rather than keeping only recent streaming data allowable in memory, because data stream mining and tracking of past data stream are often required. In the general method using wavelets, a specific amount of streaming data from a sensor is periodically compressed into fixed size and the fixed amount of compressed data is stored into stable storage. However, differently from the general method, our method flexibly adjusts the compressing size based on a heuristic criterion. Experimental results with some real stream data show that wavelets technique is useful in data stream reduction and our flexible approach has lower estimation error than the general fixed approach. © Springer-Verlag Berlin Heidelberg 2007.