Compressed prefix sums

Abstract

We consider the prefix sums problem: given a (static) sequence of positive integers x = (x1,... ,xn), such that Σ i=1ni xi = m, we wish to support the operation sum(x, j), which returns Σi=1j xii. Our interest is in minimising the space required for storing x, where 'minimal space' is defined according to some compressibility criteria, while supporting sum as rapidly as possible. There are two main compressibility criteria: (a) the succinct space bound, B(m,n) = [log2 (n-1m-1)] bits, applies to any sequence x whose elements add up to m; (b) data-aware measures, which depend on the values in x, and can be lower than the succinct bound for some sequences. Appropriate data-aware measures have been studied extensively in the information retrieval (IR) community [17]. We demonstrate a close connection between the data-aware measure that is the best in practice for an important IR application and the succinct bound. We give theoretical solutions that use space close to other data-aware compressibility measures (often within o(n) bits), and support sum in doubly-logarithmic (or better) time, and experimental evaluations of practical variants thereof. A bit-vector is a data structure that supports 'rank/select' on a bitstring, and is fundamental to succinct and compressed data structures. We describe a new bit-vector that is robust and efficient. © Springer-Verlag Berlin Heidelberg 2007.