Certainty-Based Neural Network Architecture Selection Framework for TinyML Systems

Abstract

The development of technologies related to the TinyML concept observed in recent years forces us to consider the trade-off between inference time and recognition quality. Moreover, employing data stream processing methods to analyze large volumes of data in real-time becomes necessary. In this work, considering Computer Vision data streams with a time-varying difficulty, we present a framework that allows finding a compromise between inference time and recognition quality. This trade-off can be measured using the proposed Time to Accuracy Loss measure. The presented Certainty-based Architecture Selection framework allows the selection of the neural network architecture depending on the current difficulty of the data stream. The work presents computer and TinyML device experiments. The results show that the proposed framework can effectively select the neural network architecture depending on the current difficulty of the data stream and bring benefits in the inference time.