Partitioning for Scalable Complex Event Processing on Data Streams

Abstract

Many applications processing dynamic data require to filter, aggregate, join as well as to recognize event patterns in streams of data in an online fashion. However, data analysis and complex event processing (CEP) on high volume and/or high rate streams are challenging tasks. Typically, partitioning techniques are leveraged for achieving low latency and scalable processing. Unfortunately, sequence-based operations such as CEP operations as well as long-running continuous queries make partitioning much more difficult than for batch-oriented approaches. In this paper, we address this challenge by presenting partitioning strategies for CEP queries. We discuss two strategies for stream and pattern partitioning and we present a cost-based optimization approach for determining the number of partitions as well as the split points in the queries to achieve better load balancing and avoid congestions of processing nodes in a cluster environment. © Springer International Publishing Switzerland 2015.