Efficient Incremental Learning Using Dynamic Correction Vector

Abstract

One major challenge for modern artificial neural networks (ANNs) is that they typically does not handle incremental learning well. In other words, while learning the new features, the performances of existing features usually deteriorate. This phenomenon is called catastrophic forgetting, which causes great problems for continuous, incremental, and intelligent learning. In this work, we propose a dynamic correction vector based algorithm to address both the bias problem from knowledge distillation and the overfitting problem. Specifically, we have made the following contributions: 1) we have designed a novel dynamic correction vector based algorithm; 2) we have proposed new loss functions accordingly. Experimental results on MNIST and CIFAR-100 datasets demonstrate that our technique can outperform state-of-the-art incremental learning methods by 4% on large datasets.