Approximate counting with a floating-point counter

Abstract

When many objects are counted simultaneously in large data streams, as in the course of network traffic monitoring, or Webgraph and molecular sequence analyses, memory becomes a limiting factor. Robert Morris [Communications of the ACM, 21:840-842, 1978] proposed a probabilistic technique for approximate counting that is extremely economical. The basic idea is to increment a counter containing the value X with probability 2-X . As a result, the counter contains an approximation of after n probabilistic updates, stored in bits. Here we revisit the original idea of Morris. We introduce a binary floating-point counter that combines a d-bit significand with a binary exponent, stored together on bits. The counter yields a simple formula for an unbiased estimation of n with a standard deviation of about 0.6•n2-d/2. We analyze the floating-point counter's performance in a general framework that applies to any probabilistic counter. In that framework, we provide practical formulas to construct unbiased estimates, and to assess the asymptotic accuracy of any counter. © 2010 Springer-Verlag Berlin Heidelberg.