A lightweight elastic queue middleware for distributed streaming pipeline

Abstract

We introduce an elastic queue middleware (EQM) in a distributed streaming processing architecture to handle drastically growing input streams at peak times and maintain resource utilization at off-peak times. EQM serves as a scalable stream buffer to solve bottlenecks of stream processing on the fly. With spikes in data rates, the stream buffer which holds the input tuples for a bottleneck operator scales out in EQM to immediately alleviate back pressure and the streaming engines can thus gradually deploy additional replicas of the bottleneck operator to cope with the increasing data rates. This differs from general elastic streaming processing where bottleneck operators scale out first and then the stream buffers are allocated. To implement a scalable buffer, EQM utilizes existing scalable data stores (e.g. HBase) to avoid re-inventing the same elasticity and scalability logic and meanwhile ensures load balancing performance. Experiment results show that stable throughput is achieved at varying data rates using EQM.