A Lightweight Deep Neural Network Using a Mixer-Type Nonlinear Vector Autoregression

Abstract

The design of a lightweight deep learning model would be an ideal solution for overcoming resource limitations when implementing artificial intelligence in edge sites. In this study, we propose a lightweight deep neural network that uses a Mixer-type architecture based on nonlinear vector autoregression (NVAR), which we refer to as Mixer-type NVAR. We applied overlapping patch embedding to enrich the image input and Sequencer architecture for vertical and horizontal operation inside the Mixer-type NVAR. We utilized a window partition technique and general quadratic positional encoding to increase the performance of the proposed model. Our model achieved a top-1 accuracy of 82.48% for the CIFAR-10 dataset with 0.159 M parameters and 98.36% for MNIST with 0.106 M parameters. Moreover, we evaluated its throughput on a central processing unit, which was 190.1 images per second for CIFAR-10 and 106.7 images per second for the MNIST dataset. These results are competitive with the state-of-the-art convolutional neural network-based model, MLP-Mixer, and the traditional reservoir-computing-based Mixer model with the same tuning of hyperparameters.