Distributing deep learning inference on edge devices

Abstract

Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) are widely used in IoT related applications. However, inferencing pre-trained large DNNs and CNNs consumes a significant amount of time, memory and computational resources. This makes it infeasible to use such DNNs/CNNs on resource constrained edge devices, In this research we are trying to implement a distributed inference schema for processing large DNNs and CNNs in such resource constrained edge devices. Our approach of solving this issue is based on partitioning DNNs/CNNs model and processing the inference tasks using two or more edge devices. Through this poster we introduce our novel approach of distributing the inference process among multiple edge devices while minimising the network overheads due to communication among devices. In addition to that, we present a task sharing mechanism among working devices and idle devices in the network and a way to convert pre-trained models to a separately executable model format.