On simple models of associative memory: Network density is not required for provably complex behavior

Abstract

It has been argued that complex behavior in many biological systems, including but not limited to human and animal brains, is to a great extent a consequence of high interconnectedness among the individual elements, such as neurons in brains. As a very crude approximation, brain can be viewed as an associative memory that is implemented as a large network of heavily interconnected neurons. Hopfield Networks are a popular model of associative memory. From a dynamical systems perspective, it has been posited that the complexity of possible behaviors of a Hopfield network is largely due to the aforementioned high level of interconnectedness. We show, however, that many aspects of provably complex – and, in particular, unpredictable within realistic computational resources – behavior can also be obtained in very sparsely connected Hopfield networks and related classes of Boolean Network Automata. In fact, it turns out that the most fundamental problems about the memory capacity of a Hopfield network are computationally intractable, even for restricted types of networks that are uniformly sparse, with only a handful neighbors per node. One implication of our results is that some of the most fundamental aspects of biological (and other) networks’ dynamics do not require high density, in order to exhibit provably complex, computationally intractable behavior.