Dynamic and coordinated software reconfiguration in distributed data stream systems

Abstract

While many systems have to provide 24 × 7 services with no acceptable downtime, they have to be able to cope with changes in their execution environment and in the requirements that they must comply, in which data stream processing is one example of system that has to evolve during its execution. On one hand, dynamic reconfiguration (i.e., the capability of evolving on-the-fly) is a desirable feature. On the other hand, stream systems may suffer with the disruption and overhead caused by the reconfiguration. Due to these conflicting requirements, safe and non-disruptive reconfiguration is still an open problem. In this paper, we propose and validate a non-disruptive reconfiguration approach for distributed data stream systems that support stateful components and intermittent connections. We present experimental evidence that our mechanism supports safe distributed reconfiguration and has negligible impact on availability and performance.