Relating sparse and predictive coding to divisive normalization

Abstract

Sparse coding, predictive coding and divisive normalization have each been found to be principles that underlie the function of neural circuits in many parts of the brain, supported by substantial experimental evidence. However, the connections between these related principles are still poorly understood. Sparse coding and predictive coding can be reconciled into a learning framework with predictive structure and sparse responses, termed as sparse/predictive coding. However, how sparse/predictive coding (a learning model) is connected with divisive normalization (not a learning model) is still not well investigated. In this paper, we show how sparse coding, predictive coding, and divisive normalization can be described within a unified framework, and illustrate this explicitly within the context of a two-layer neural learning model of sparse/predictive coding. This two-layer model is constructed in a way that implements sparse coding with a network structure that is constructed by implementing predictive coding. We demonstrate how a homeostatic function that regulates neural responses in the model can shape the nonlinearity of neural responses in a way that replicates different forms of divisive normalization. Simulations show that the model can learn simple cells in the primary visual cortex with the property of contrast saturation, which has previously been explained by divisive normalization. In summary, the study demonstrates that the three principles of sparse coding, predictive coding, and divisive normalization can be connected to provide a learning framework based on biophysical properties, such as Hebbian learning and homeostasis, and this framework incorporates both learning and more diverse response nonlinearities observed experimentally. This framework has the potential to also be used to explain how the brain learns to integrate input from different sensory modalities.