View source for Information theory and measure theory

{{nofootnotes|date=December 2023}}
This article discusses how [[Information theory|information theory]] (a branch of mathematics studying the transmission, processing and storage of [[Information|information]]) is related to measure theory (a branch of mathematics related to [[Integral|integration]] and [[Probability|probability]]).

== Measures in information theory ==
Many of the concepts in information theory have separate definitions and formulas for continuous and discrete cases. For example, [[Physics:Entropy|entropy]] <math>\Eta(X)</math> is usually defined for discrete random variables, whereas for continuous random variables the related concept of [[Differential entropy|differential entropy]], written <math>h(X)</math>, is used (see Cover and Thomas, 2006, chapter 8). Both these concepts are mathematical [[Expected value|expectations]], but the expectation is defined with an [[Integral|integral]] for the continuous case, and a sum for the discrete case.

These separate definitions can be more closely related in terms of measure theory.  For discrete random variables, probability mass functions can be considered density functions with respect to the counting measure. Thinking of both the integral and the sum as integration on a measure space allows for a unified treatment. 

Consider the formula for the differential entropy of a continuous [[Random variable|random variable]] <math>X</math> with range <math>\mathbb{R}</math> and [[Physics:Probability density function|probability density function]] <math>f(x)</math>:
:<math> h(X) = -\int_\mathbb{R} f(x) \log f(x) \,dx. </math>
This can usually be interpreted as the following [[Riemann–Stieltjes integral]]:
:<math> h(X) = -\int_\mathbb{R} f(x) \log f(x) \,d\mu(x), </math>
where <math>\mu</math> is the [[Lebesgue measure]].

If instead, <math>X</math> is discrete, with range <math>\Omega</math> a finite set, <math>f</math> is a probability mass function on <math>\Omega</math>, and <math>\nu</math> is the [[Counting measure|counting measure]] on <math>\Omega</math>, we can write:
:<math> \Eta(X) = -\sum_{x\in \Omega} f(x) \log f(x) = -\int_\Omega f(x) \log f(x) \,d\nu(x). </math> 
The integral expression, and the general concept, are identical in the continuous case; the only difference is the measure used.  In both cases the probability density function <math>f</math> is the [[Radon–Nikodym theorem|Radon–Nikodym derivative]] of the [[Probability measure|probability measure]] with respect to the measure against which the integral is taken.

If <math> P </math> is the probability measure induced by <math>X</math>, then the integral can also be taken directly with respect to <math> P </math>:
:<math> h(X) = -\int_{\Omega} \log \frac{\mathrm d P}{\mathrm d\mu} \,dP, </math>

If instead of the underlying measure μ we take another probability measure <math> Q </math>, we are led to the [[Kullback–Leibler divergence]]: let <math>P</math> and <math>Q</math> be probability measures over the same space.  Then if <math>P</math> is [[Absolute continuity|absolutely continuous]] with respect to <math>Q</math>, written <math>P \ll Q,</math> the [[Radon–Nikodym theorem|Radon–Nikodym derivative]] <math>\frac{\mathrm dP}{\mathrm dQ}</math> exists and the Kullback–Leibler divergence can be expressed in its full generality:

:<math>D_\operatorname{KL}(P \| Q)
   = \int_{\operatorname{supp}P}
     \frac{\mathrm dP}{\mathrm dQ}
     \log \frac{\mathrm dP}{\mathrm dQ}
     \,d Q
   = \int_{\operatorname{supp}P}
     \log \frac{\mathrm dP}{\mathrm dQ}
       \,d P,
</math>
where the integral runs over the [[Support (mathematics)|support]] of <math>P.</math>  Note that we have dropped the negative sign: the Kullback–Leibler divergence is always non-negative due to [[Gibbs' inequality]].

==Entropy as a "measure"==

[[Image:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram|Venn diagram]] for various information measures associated with correlated variables ''X'' and ''Y''. The area contained by both circles is the joint entropy ''H''(''X'',''Y''). The circle on the left (red and cyan) is the individual entropy ''H''(''X''), with the red being the conditional entropy ''H''(''X''&#124;''Y''). The circle on the right (blue and cyan) is ''H''(''Y''), with the blue being ''H''(''Y''&#124;''X''). The cyan is the mutual information ''I''(''X'';''Y'').]]{{Refimprove section|date=April 2017}}[[Image:VennInfo3Var.svg|thumb|256px|left|[[Venn diagram|Venn diagram]] of information theoretic measures for three variables ''x'', ''y'', and ''z''. Each circle represents an individual [[Entropy (information theory)|entropy]]: ''H''(''x'') is the lower left circle, ''H''(''y'') the lower right, and ''H''(''z'') is the upper circle. The intersections of any two circles represents the [[Mutual information|mutual information]] for the two associated variables (e.g. ''I''(''x'';''z'') is yellow and gray). The union of any two circles is the [[Joint entropy|joint entropy]] for the two associated variables (e.g. ''H''(''x'',''y'') is everything but green). The joint entropy ''H''(''x'',''y'',''z'') of all three variables is the union of all three circles. It is partitioned into 7 pieces, red, blue, and green being the [[Conditional entropy|conditional entropies]] ''H''(''x''&#124;''y'',''z''), ''H''(''y''&#124;''x'',''z''), ''H''(''z''&#124;''x'',''y'') respectively, yellow, magenta and cyan being the [[Conditional mutual information|conditional mutual informations]] ''I''(''x'';''z''&#124;''y''), ''I''(''y'';''z''&#124;''x'') and ''I''(''x'';''y''&#124;''z'') respectively, and gray being the [[Multivariate mutual information|multivariate mutual information]] ''I''(''x'';''y'';''z''). The multivariate mutual information is the only one of all that may be negative.]]

There is an analogy between Shannon's basic "[[Quantities of information|measures]]" of the information content of random variables and a measure over sets.  Namely the [[Joint entropy|joint entropy]], [[Conditional entropy|conditional entropy]], and [[Mutual information|mutual information]] can be considered as the measure of a [[Union (set theory)|set union]], set difference, and [[Intersection (set theory)|set intersection]], respectively (Reza pp.&nbsp;106–108).

If we associate the existence of abstract [[Set (mathematics)|sets]] <math>\tilde X</math> and <math>\tilde Y</math> to arbitrary [[Discrete mathematics|discrete]] [[Random variable|random variable]]s ''X'' and ''Y'', somehow representing the information borne by ''X'' and ''Y'', respectively, such that:
* <math>\mu(\tilde X \cap \tilde Y) = 0 </math> whenever ''X'' and ''Y'' are unconditionally independent, and
* <math>\tilde X = \tilde Y</math> whenever ''X'' and ''Y'' are such that either one is completely determined by the other (i.e. by a bijection);

where <math>\mu</math> is a [[Signed measure|signed measure]] over these sets, and we set:

: <math>
\begin{align}
\Eta(X) & = \mu(\tilde X), \\
\Eta(Y) & = \mu(\tilde Y), \\
\Eta(X,Y) & = \mu(\tilde X \cup \tilde Y), \\
\Eta(X\mid Y) & = \mu(\tilde X \setminus \tilde Y), \\
\operatorname{I}(X;Y) & = \mu(\tilde X \cap \tilde Y);
\end{align}
</math>

we find that Shannon's "measure" of information content satisfies all the postulates and basic properties of a formal [[Signed measure|signed measure]] over sets, as commonly illustrated in an '''[[Information diagram|information diagram]]'''. This allows the sum of two measures to be written:

:<math>
\mu(A)+\mu(B)=\mu(A\cup B)+\mu(A\cap B)
</math>

and the analog of [[Bayes' theorem]] (<math>\mu(A)+\mu(B\setminus A)=\mu(B)+\mu(A\setminus B)</math>) allows the difference of two measures to be written:
:<math>
\mu(A)-\mu(B)=\mu(A\setminus B)-\mu(B\setminus A)
</math>

This can be a handy mnemonic device in some situations, e.g.

: <math>
\begin{align}
\Eta(X,Y) & =\Eta(X)+\Eta(Y\mid X) & \mu(\tilde X \cup \tilde Y) & =\mu(\tilde X)+\mu(\tilde Y \setminus \tilde X) \\
\operatorname{I}(X;Y) & =\Eta(X)-\Eta(X\mid Y) & \mu(\tilde X \cap \tilde Y) & =\mu(\tilde X)-\mu(\tilde X \setminus \tilde Y)
\end{align}
</math>

Note that measures (expectation values of the logarithm) of true probabilities are called "entropy" and generally represented by the letter ''H'', while other measures are often referred to as "information" or "correlation" and generally represented by the letter ''I''. For notational simplicity, the letter ''I'' is sometimes used for all measures.

==Multivariate mutual information==
{{main|Multivariate mutual information}}
Certain extensions to the definitions of Shannon's basic measures of information are necessary to deal with the [[Sigma-algebra|σ-algebra]] generated by the sets that would be associated to three or more arbitrary random variables. (See Reza pp.&nbsp;106–108 for an informal but rather complete discussion.) Namely <math>\Eta(X,Y,Z,\cdots)</math> needs to be defined in the obvious way as the entropy of a joint distribution, and a multivariate [[Mutual information|mutual information]] <math>\operatorname{I}(X;Y;Z;\cdots)</math> defined in a suitable manner so that we can set:

: <math>
\begin{align}
\Eta(X,Y,Z,\cdots) & = \mu(\tilde X \cup \tilde Y \cup \tilde Z \cup \cdots), \\
\operatorname{I}(X;Y;Z;\cdots) & = \mu(\tilde X \cap \tilde Y \cap \tilde Z \cap \cdots);
\end{align}
</math>

in order to define the (signed) measure over the whole σ-algebra.  There is no single universally accepted definition for the multivariate mutual information, but the one that corresponds here to the measure of a set intersection is due to Fano (1966: p. 57-59). The definition is recursive.  As a base case the mutual information of a single random variable is defined to be its entropy: <math>\operatorname{I}(X)=\Eta(X)</math>.  Then for <math>n\geq 2</math> we set
: <math>\operatorname{I}(X_1; \cdots; X_n) = \operatorname{I}(X_1; \cdots; X_{n-1}) - \operatorname{I}(X_1; \cdots; X_{n-1}\mid X_n),</math>
where the [[Conditional mutual information|conditional mutual information]] is defined as
: <math>\operatorname{I}(X_1; \cdots; X_{n-1}\mid X_n) = \mathbb E_{X_n}\big(\operatorname{I}(X_1; \cdots; X_{n-1})\mid X_n\big).</math>
The first step in the recursion yields Shannon's definition <math>\operatorname{I}(X_1;X_2)=\Eta(X_1)-\Eta(X_1\mid X_2).</math>  The multivariate mutual information (same as [[Interaction information|interaction information]] but for a change in sign) of three or more random variables can be negative as well as positive: Let ''X'' and ''Y'' be two independent fair coin flips, and let ''Z'' be their [[Exclusive or|exclusive or]]. Then <math>\operatorname{I}(X;Y;Z) = - 1</math> bit.

Many other variations are possible for three or more random variables: for example, <math>\operatorname{I}(X,Y;Z)</math> is the mutual information of the joint distribution of ''X'' and ''Y'' relative to ''Z'', and can be interpreted as <math>\mu((\tilde X \cup \tilde Y) \cap \tilde Z).</math>  Many more complicated expressions can be built this way, and still have meaning, e.g. <math> \operatorname{I}(X,Y;Z\mid W),</math> or <math> \Eta(X,Z\mid W,Y).</math>

==References==
* Thomas M. Cover and Joy A. Thomas. ''Elements of Information Theory'', second edition, 2006. New Jersey: Wiley and Sons. {{ISBN|978-0-471-24195-9}}.
* Fazlollah M. Reza. ''An Introduction to Information Theory''. New York: McGraw–Hill 1961. New York: Dover 1994. {{ISBN|0-486-68210-2}}
* {{citation | last=Fano | first=R. M. |  title=Transmission of Information: a statistical theory of communications | publisher= MIT Press | year=1966 | url=https://archive.org/details/TransmissionOfInformationAStatisticalTheoryOfCommunicationRobertFano| isbn=978-0-262-56169-3 | oclc=804123877 }}
* R. W. Yeung, "On entropy, information inequalities, and Groups." [http://user-www.ie.cuhk.edu.hk/~ITIP/online/tutorial.ps PS]

==See also==
*[[Information theory]]
*Measure theory
*[[Set theory]]

[[Category:Information theory]]
[[Category:Measure theory]]

{{Sourceattribution|Information theory and measure theory}}