Model Training Task Scheduling Algorithm Based on Greedy-Genetic Algorithm for Big-Data Mining

Abstract

With the coming of the big data age, data mining attracts more and more attention from all trades and professions. Due to the vast computation cost of data mining, the public service platform for big data mining has become the urgent needs, especially for the model training tasks. In this way, how to perform this kind of task scheduling becomes critical. This paper focuses on the assignment of tasks on multiple computing resources to optimize the total operation time. Firstly, a task scheduling algorithm based on the greedy and genetic algorithm is proposed to set the computation resource requirement for each task. Moreover, a greedy strategy is used to decide the task operation order and the assignment mapping between the tasks and the computation resources. Finally, the proposed algorithm proves to be efficient by several experiments.