'Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory.' Ann. We use this theory to introduce a new concept of “generalized exponential family” linked to the specific decision problem under consideration, and we demonstrate that this shares many of the properties of standard exponential families.įinally, we show that the existence of an equilibrium in our game can be rephrased in terms of a “Pythagorean property” of the related divergence, thus generalizing previously announced results for Kullback–Leibler and Bregman divergences.

This leads to an extension, to arbitrary discrepancies, of a well-known minimax theorem for the case of Kullback–Leibler divergence (the “redundancy-capacity theorem” of information theory).įor the important case of families of distributions having certain mean values specified, we develop simple sufficient conditions and methods for identifying the desired solutions. We also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions. This simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust Bayes acts. We indicate how an appropriate generalized definition of entropy can be associated with such a problem, and we show that, subject to certain regularity conditions, the above-mentioned duality continues to apply in this extended context. We here generalize this theory to apply to arbitrary decision problems and loss functions. The proposed principle of maximum entropy production (MEP), based on statistical mechanics and information theory, states that thermodynamic processes far from thermodynamic equilibrium will adapt to steady states at which they dissipate energy and produce entropy at the maximum possible rate. (This is equivalent to an alternate definition for the term entropy as a measure of the disorder of a system, such that a completely random dispersion of elements corresponds to maximum entropy, or minimum information.

Although Topsøe described this connection for the Shannon entropy over 20 years ago, it does not appear to be widely known even in that important special case. In general, the equilibrium state for an isolated system is precisely that state of maximum entropy. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to the other. We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss.

Maximum entropy