Reprinted from THE PHYSICAL REVIEW, Vol. 106, No.4, 620-630, May IS, 1957
Printed in U. S. A.
Information Theory and Statistical Mechanics
E. T. JAYNES
Department of Physics, Stanford University, Stanford, CaJifornia
(Received September 4, 1956; revised manuscript received March 4, 1957)
Information theory provides a constructive criterion for setting
up probability distributions on the basis of partial knowledge,
and leads to a type of statistical inference which is called the
maximum-entropy estimate. It is the least biased estimate
possible on the given information; i.e., it is maximally noncom›
mittal with regard to missing information. If one considers
statistical mechanics as a form of statistical inference rather than
as a physical theory, it is found that the usual computational
rules, starting with the determination of the partition function,
are an immediate consequence of the maximum-entropy principle.
In the resulting "subjective statistical mechanics," the usual rules
are thus justified independently of any physical argument, and
in particular independently of experimental verification; whether
1. INTRODUCTION
T
HE recent appearance of a very comprehensive
survey! of past attempts to justify the methods
of statistical mechanics in terms of mechanics, classical
or quantum, has helped greatly, and at a very opportune
time, to emphasize the unsolved problems in this field.
1 D. ter Haar, Revs. Modern Phys. 27, 289 (1955).
or not the results agree with experiment, they still represent the
best estimates that could have been made on the basis of the
information available.
It is conCluded that statistical mechanics need not be regarded
as a physical theory dependent for its validity on the truth of
additional assumptions not contained in the laws of mechanics
(such as ergodicity, metric transitivity, equal a priori probabilities,
etc.). Furthermore, it is possible to maintain a sharp distinction
between its physical and statistical aspects. The former consists
only of the correct enumeration of the states of a system and
their properties; the latter is a straightforward example of
statistical inference.
Although the subject has been under development for
many years, we still do not have a complete and
satisfactory theory, in the sense that there is no line
of argument proceeding from the laws of microscopic
mechanics to macroscopic phenomena, that is generally
regarded by physicists as convincing in all respects.
Such an argument should (a) be free from objection on
mathematical grounds, (b) involve no additional arbi-

621 IN FOR MAT ION THE 0 R Y AND S TAT 1ST I CAL ME C HAN I C S
trary assumptions, and (c) automatically include an
explanation of nonequilibrium conditions and irre›
versible processes as well as those of conventional
thermodynamics, since equilibrium thermodynamics is
merely an ideal limiting case of the behavior of matter.
It might appear that condition (b) is too severe,
since we expect that a physical theory will involve
certain unproved assumptions, whose consequences are
deduced and compared with experiment. For example,
in the statistical mechanics of Gibbs
2
there were several
difficulties which could not be understood in terms of
classical mechanics, and before the models which he
constructed co uld be made to correspond to the observed
facts, it was necessary to incorporate into them addi›
tional restrictions not contained in the laws of classical
mechanics . First was the "freezing up" of certain
degrees of freedom, which caused the specific heat of
diatomic gases to be only i of the expected value.
Secondly, the paradox regarding the entropy of com›
bined systems, which was resolved only by adoption of
the generic instead of the specific definition of phase,
an assumption which seems impossible to justify in
terms of classical notions.
3
Thirdly, in order to account
for the actual values of vapor pressures and equilibrium
constants, an additional assumption about a natural
unit of volume (h
3N
) of phase space was needed.
However, with the development of quantum mechanics
the originally arbitrary assumptions are now seen as
necessary consequences of the laws of physics. This
suggests the possibility that we have now reached a
state where statistical mechanics is no longer dependent
on physical hypotheses, but may become merely an
example of statistical inference.
That the present may be an opportune time to
re-examine these questions is due to two recent de›
velopments. Statistical methods are being applied to a
variety of specific phenomena involving irreversible
processes, and the mathematical methods which have
proven successful have not yet been incorporated into
the basic apparatus of statistical mechanics. In addition,
the development of information theory! has been felt
by many people to be of great significance for statistical
mechanics, although the exact way in which it should
be applied has remained obscure. In this connection it
2 J. W. Gibbs, Elementary Principles in Statistical Mechanics
(Longmans Green and Company, New York, 1928), Vol. II of
collected works.
3 We may note here that although Gibbs (reference 2, Chap.
XV) started his discussion of this question by saying that the
generic definition "seems in accordance with the spirit of the
statistical method," he concluded it with, "The perfect similarity
of several particles of a system will not in the lc:ast interfere with
the identification of a particular particle in one case with a
particular particle in another. The question is one to be decided
in accordance with the requirements of practical convenience in
the discussion of the problems with which we are engaged."
’C. E. Shannon, Bell System Tech. J. 27, 379, 623 (1948);
these papers are reprinted in C. E. Shannon and W. Weaver,
The Mathematical Theory of Communication (University of
Illinois Press, Urbana, 1949).
is essential to note the following. The mere fact that
the same mathematical expression - L Pi log Pi occurs
both in statistical mechanics and in information theory
does not in itself establish any connection between
these fields. This can be done only by finding new
viewpoints from which thermodynamic entropy and
information-theory entropy appear as the same concept.
In this paper we suggest a reinterpretation of statistical
mechanics which accomplishes this, so that information
theory can be applied to the problem of justification of
statistical mechanics. We shall be concerned with the
prediction of equilibrium thermodynamic properties,
by an elementary treatment which involves only the
probabilities assigned to stationary states. Refinements
obtainable by use of the density matrix and discussion
of irreversible processes will be taken up in later papers.
Section 2 defines and establishes some of the ele›
mentary properties of maximum-entropy inference, and
in Sees. 3 and 4 the application to statistical mechanics
is discussed. The mathematical facts concerning maxi›
mization of entropy, as given in Sec. 2, were pointed
out long ago by Gibbs. In the past, however, these
properties were given the status of side remarks not
essential to the theory and not providing in themselves
any justification for the methods of statistical me›
chanics. The feature which was missing has been
supplied only recently by Shannon
4
in the demon›
stration that the expression for entropy has a deeper
meaning, quite independent of thermodynamics. This
makes possible a reversal of the usual line of reasoning in
statistical mechanics. Previously, one constructed a
theory based on the equations of motion, supplemented
by additional hypotheses of ergodicity, metric transi›
tivity, or equal a priori probabilities, and the identifi›
cation of entropy was made only at the end, by com›
parison of the resulting equations with the laws of
phenomenological thermodynamics. Now, however, we
can take entropy as our starting concept, and the fact
that a probability distribution maximizes the entropy
subject to certain constraints becomes the essential fact
which justifies use of that distribution for inference.
The most important consequence of this reversal of
viewpoint is not, however, the conceptual and mathe›
matical simplification which results. In freeing the
theory from its apparent dependence on physical
hypotheses qf the above type, we make it possible to
see statistical mechanics in a much more general light.
Its principles and mathematical methods become
available for treatment of many new physical problems.
Two examples are provided by the derivation of Siegert’s
"pressure ensemble" and treatment of a nuclear polari›
zation effect, in Sec. 5.
2. MAXIMUM-ENTROPY ESTIMATES
The quantity x is capable of assuming the discrete
values Xi (i = 1,2 ... ,n). Weare not given the corre›
sponding probabilities Pi; all we know is the expectation