Hybrid workflow scheduling on edge cloud computing systems

Abstract

Internet of Things applications can be represented as workflows in which stream and batch processing are combined to accomplish data analytics objectives in many application domains such as smart home, health care, bioinformatics, astronomy, and education. The main challenge of this combination is the differentiation of service quality constraints between batch and stream computations. Stream processing is highly latency-sensitive while batch processing is more likely resource-intensive. In this work, we propose an end-to-end hybrid workflow scheduling on an edge cloud system as a two-stage framework. In the first stage, we propose a resource estimation algorithm based on a linear optimization approach, gradient descent search (GDS), and in the second stage, we propose a cluster-based provisioning and scheduling technique for hybrid workflows on heterogeneous edge cloud resources. We provide a multi-objective optimization model for execution time and monetary cost under constraints of deadline and throughput. Results demonstrate the framework performance in controlling the execution of hybrid workflows by efficiently tuning several parameters including stream arrival rate, processing throughput, and workflow complexity. In comparison to a meta-heuristics technique using Particle Swarm Optimization (PSO), the proposed scheduler provides significant improvement for large-scale hybrid workflows in terms of execution time and cost with an average of 8% and 35%, respectively.