Estimating frequency moments of data streams using random linear combinations

Abstract

The problem of estimating the kth frequency moment Fk for any nonnegative k, over a data stream by looking at the items exactly once as they arrive, was considered in a seminal paper by Alon, Matias and Szegedy [1,2]. The space complexity of their algorithm is Õ(n1-1/k). For k > 2, their technique does not apply to data streams with arbitrary insertions and deletions. In this paper, we present an algorithm for estimating Fk for k > 2, over general update streams whose space complexity is Õ(n1-1/k-1) and time complexity of processing each stream update is Õ(1). Recently, an algorithm for estimating Fk over general update streams with similar space complexity has been published by Coppersmith and Kumar [7]. Our technique is, (a) basically different from the technique used by [7], (b) is simpler and symmetric, and, (c) is more efficient in terms of the time required to process a stream update (Õ(1) compared with (Õ(n1-1/k-1)). © Springer-Verlag 2004.