Efficient DNN Execution on Intermittently-Powered IoT Devices with Depth-First Inference

Abstract

Program execution on intermittently powered Internet-of-Things (IoT) devices must ensure forward progress in the presence of frequent power failures. A general solution is intermittent computing, by which the program states are frequently checkpointed to non-volatile memory (NVM) so that once a power failure occurs, the program can restart from the latest checkpoint after the system energy regains. However, executing a deep neural network (DNN) inference program in an intermittent way has a big problem. During the execution, an inference program will generate large-volume feature maps (as part of the program states), and checkpointing the feature maps to NVM will incur significant time and energy overhead and thus reduce the inference efficiency. This paper proposes an approach to reduce the amount of feature map writing in intermittent DNN inference. The main idea is to partition the inference task into several slices and execute each slice in a depth-first manner so that the intermediate feature maps during the inference of each slice do not need to be written to NVM. Extensive experiments have been conducted, which show that the proposed approach can significantly reduce the amount of NVM writing, and a maximum of 1.965 speedup of the total inference time is achieved compared to the state-of-the-art approach.