Neighborhood search-based job scheduling for IoT big data real-time processing in distributed edge-cloud computing environment

Abstract

Cloud-edge collaboration architecture, which combines edge processing and centralized cloud processing, is suitable for placement and caching of streaming media. A cache-aware scheduling model based on neighborhood search is proposed. The model is divided into four sub-problems: job classification, node resource allocation, node clustering, and cache-aware job scheduling. Firstly, jobs are categorized into three categories, and then different resources are allocated to nodes according to different job execution conditions. Secondly, the nodes with similar capabilities are clustered, and the jobs are cached by delay-waiting. For jobs that do not satisfy the data locality, the jobs are scheduled to the nodes with similar capabilities according to the neighborhood search results. Meanwhile, a cache-aware scheduling algorithm based on neighborhood search is proposed. Experiments show that the proposed algorithm can effectively minimize the delay of content transmission and the cost of content placement, the job execution time is shortened and the processing capacity of the cloud data center is improved.