Intermittent Inference: Trading a 1% Accuracy Loss for a 1.9x Throughput Speedup

Abstract

We present INTERCEPT, a compile-time toolchain enabling manifold throughput improvements when running intermittent DNN inference on IoT devices, in exchange of a maximum 1% accuracy loss. Intermittently-computing IoT devices rely on ambient energy harvesting and compute opportunistically, as energy is available. They use NVM to persist intermediate results in anticipation of energy failures. Without requiring changes to existing models and by exploiting the features of STT-MRAM as NVM, INTERCEPT optimizes the placement and configuration of state persistence operations when executing the inference process. This happens off-line with no user intervention, while enforcing a maximum 1% accuracy loss. Our results, obtained across three platforms and six diverse neural networks, indicate that INTERCEPT provides a 40% energy gain in a single inference process, on average. With the same energy budget, this yields a 1.9x throughput speedup.