Low-level Optimizations for Faster Mobile Deep Learning Inference Frameworks

Abstract

Over the last ten years, we have seen a strong progression of technology around smartphones. Each new generation acquires capabilities that significantly increase performance. On the other hand, several deep learning tools are offered today by the giants of the net for mobile, embedded devices and IoT. The proposed libraries allow a machine learning inference on the device with low latency. They provide pre-trained models, but one can also use one's own models and run them on mobile, embedded or microcontroller devices. Lack of privacy, poor Internet connectivity and high cost of cloud platform let on-device inference became popular through app developers but there are more significant challenges especially for real-time tasks like augmented reality or autonomous driving. This PhD research aims at providing a path for developers to help them choose the best methods and tools to do real-time inference on mobile devices. In this paper, we present the performance benchmark of four popular open-source deep learning inference frameworks used on mobile devices on three different convolutional neural network models. We focus our work on image classification process and particularly on validation image bank of ImageNet 2012 dataset. We try to answer three questions : How does a framework influence model prediction and latency - Why some frameworks are better in terms of latency/accuracy than others with the same model - And what are the difficulties to implement these frameworks inside a mobile application - Our first findings demonstrate that low-level software implementations chosen in frameworks, model conversion steps and parameters set in the framework have a big impact on performance and accuracy.