Bio-inspired Stochastic Growth and Initialization for Artificial Neural Networks

Abstract

Current initialization methods for artificial neural networks (ANNs) assume full connectivity between network layers. We propose that a bio-inspired initialization method for establishing connections between neurons in an artificial neural network will produce more accurate results relative to a fully connected network. We demonstrate four implementations of a novel, stochastic method for generating sparse connections in spatial, growth-based connectivity (GBC) maps. Connections in GBC maps are used to generate initial weights for neural networks in a deep learning compatible framework. These networks, designated as Growth-Initialized Neural Networks (GrINNs), have sparse connections between the input layer and the hidden layer. GrINNs were tested with user-specified nominal connectivity percentages ranging from 5–45%, resulting in unique connectivity percentages ranging from 4–28%. For reference, fully connected networks are defined as having 100% unique connectivity within this context. GrINNs with nominal connectivity percentages 20% produced better accuracy than fully connected ANNs when trained and tested on the MNIST dataset.