Quantization error-based regularization in neural networks

Abstract

Deep neural network is a state-of-the-art technology for achieving high accuracy in various machine learning tasks. Since the available computing power and memory footprint are restricted in embedded computing, precision quantization of numerical representations, such as fixed-point, binary, and logarithmic, are commonly used for higher computing efficiency. The main problem of quantization is accuracy degradation due to its lower numerical representation. There is generally a trade-off between numerical precision and accuracy. In this paper, we propose a quantization-error-aware training method to attain higher accuracy in quantized neural networks. Our approach appends an additional regularization term that is based on quantization errors of weights to the loss function. We evaluate the accuracy by using MNIST and CIFAR-10. The evaluation results show that the proposed approach achieves higher accuracy than the standard approach with quantized forwarding.