A Pipeline-Based Scheduler for Optimizing Latency of Convolution Neural Network Inference over Heterogeneous Multicore Systems

Abstract

Parallelization is a common design practice for throughput improvement on multicore systems. However, the existing operating systems' schedulers for CNN inference essentially divide the computational tasks of each convolution layer onto different CPU cores and cause significant inter-core feature-map data movement. Therefore, the overall performance is often degraded. In this paper, we propose a pipeline-based scheduler for convolution neural network inference parallelization with minimal feature-map data movement requirements. The experimental results show that our approach can achieve 73% performance improvement on throughput compared to the existing multi-thread scheduler.