Business Intelligence and Big Data

Abstract

Sometimes data is generated unboundedly and at such a fast pace that it is no longer possible to store the complete data in a database. The development of techniques for handling and processing such streams of data is very challenging as the streaming context imposes severe con- straints on the computation: we are often not able to store the whole data stream and making multiple passes over the data is no longer pos- sible. As the stream is never finished we need to be able to continuously provide, upon request, up-to-date answers to analysis queries. Even prob- lems that are highly trivial in an off-line context, such as: “How many different items are there in my database?” become very hard in a stream- ing context. Nevertheless, in the past decades several clever algorithms were developed to deal with streaming data. This paper covers several of these indispensable tools that should be present in every big data sci- entists’ toolbox, including approximate frequency counting of frequent items, cardinality estimation of very large sets, and fast nearest neigh- bor search in huge data collections. 1