Convolutional Neural Networks with Hebbian-Based Rules in Online Transfer Learning

Abstract

In 1949, Donald Hebb proposed its Neurophysiologic Principle, which models the weight change of connected neurons. Although similar mechanisms have been experimentally proven to exists in several brain areas, traditional Deep Learning methods do not implement it, generally using Gradient-based algorithms instead. On the other hand, Convolutional Neural Networks (CNNs) has slight inspiration on the structure of Visual Cortex, particularly on the Hierarchical model of Hubel-Wiesel. Using convolutional layers it is possible to perform feature extraction and a final classification with dense layers. In this paper, we propose a combined technique of using pre-trained convolutional layers and a final classification using Hebbian-based rules (Basic Hebb, Covariance, Oja, and BCM). Once the feature extraction is done, Hebbian rules can discriminate the classes with high accuracy. These theoretical ideas were tested using this MNIST database, reaching 99.43% of test accuracy training CNNs layers with a final Hebb layer. Similar results were found using Hebbian-based rules in other datasets with RGB images and Transfer Learning. Even when these results are slightly lower than Gradient-based methods, Hebbian learning can perform online learning, which suggests that this combined strategy might be useful to design Online Machine Learning Algorithms for Image Classification.