Neural Computation from First Principles: Using the Maximum Entropy Method to Obtain an Optimal Bits-Per-Joule Neuron

Abstract

Optimization results are one method for understanding neural computation from nature's perspective and for defining the physical limits on neuron-like engineering. Earlier work looks at individual properties or performance criteria and occasionally a combination of two, such as energy and information. Here, as the optimization method, we make use of Jaynes' maximum entropy method and point out some of the different types of constraints, possibly dimensionally distinct, the method can combine. A neuron, as a computational device, is assumed to estimate a scalar latent variable and to encode this estimate with an interpulse-interval. Arising from each such constraint set, the inference method identifies a likelihood-function and a sufficient statistic for the particular estimation problem. This likelihood is a first-hitting time distribution in the exponential family. Particular constraint sets are identified that, from an optimal inference perspective, align with earlier neurocomputational models. Interactions between constraints, mediated through the inferred likelihood, restrict constraint-set parameterizations, e.g., the energy-budget limits action potential threshold which limits estimation performance. Such linkages are, for biologists, experimental predictions arising from the method. In addition to the likelihood, which is a conditional distribution of the interpulse interval given the variable being estimated, at least one type of constraint set restricts the two marginal distributions. In this case, a Shannon bits/joule statement arises using Lindley's interpretation of a Bayesian experiment to get bits per pulse-out.