Everything Everyway All at Once -Time Traveling Debugging for Stream Processing Applications

Abstract

Debugging, evaluating, and optimizing stream processing applications is challenging due to continuous streams of input data and typically distributed and parallel execution environments. To address these issues, we present an approach for explorative debugging of stream processing pipelines that allows in-depth investigation of a pipeline's execution behavior and evolution. The time traveling debugger enables traveling back in time within the pipeline's execution history and thoroughly analyzing and retracing each fine-grained step. Any changes made to the pipeline's structure or parameters are captured based on provenance information and can be reviewed, compared, and analyzed with the provenance inspector to understand the impact of each alteration on the quality of the pipeline.