Optimal matrix compression yields storage capacity 1 for binary Willshaw associative memory

Abstract

The classical binary Willshaw model of associative memory has an asymptotic storage capacity of In 2 ≈ 0.7 which exceeds the capacities of other (e.g., Hopfield-like) models by far. However, its practical use is severely limited, since the asymptotic capacity is reached only for very large numbers n of neurons and for sparse patterns where the number k of one-entries must match a certain optimal value kopt(n) (typically kopt = log n). In this work I demonstrate that optimal compression of the binary memory matrix by a Huffman or Golomb code can increase the asymptotic storage capacity to 1. Moreover, it turns out that this happens for a very broad range of k being either ultra-sparse (e.g., k constant) or moderately-sparse (e.g., k = √n). A storage capacity in the range of ln 2 is already achieved for practical numbers of neurons. © Springer-Verlag Berlin Heidelberg 2003.