View source for Joint entropy

{{Short description|Measure of information in probability and information theory}}
{{Information theory}}

[[Image:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|A misleading<ref>{{Cite book|author=D.J.C. Mackay|title= Information theory, inferences, and learning algorithms|year= 2003|bibcode= 2003itil.book.....M}}{{rp|141}}</ref> [[Venn diagram|Venn diagram]] showing additive, and subtractive relationships between various [[Quantities of information|information measures]] associated with correlated variables X and Y. The area contained by both circles is the joint entropy H(X,Y). The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] H(X), with the red being the [[Conditional entropy|conditional entropy]] H(X&#124;Y). The circle on the right (blue and violet) is H(Y), with the blue being H(Y&#124;X). The violet is the [[Mutual information|mutual information]] I(X;Y).]]

In [[Information theory|information theory]], '''joint [[Entropy (information theory)|entropy]]''' is a measure of the uncertainty associated with a set of variables.<ref name=korn>{{cite book |author1=Theresa M. Korn|author2=Korn, Granino Arthur |title=Mathematical Handbook for Scientists and Engineers: Definitions, Theorems, and Formulas for Reference and Review |date= January 2000 |publisher=Dover Publications |location=New York |isbn=0-486-41147-8 }}</ref>

==Definition==
The joint Shannon entropy (in [[Bit|bit]]s) of two discrete [[Random variable|random variables]] <math>X</math> and <math>Y</math> with images <math>\mathcal X</math> and <math>\mathcal Y</math> is defined as<ref name=cover1991>{{cite book |author1=Thomas M. Cover |author2=Joy A. Thomas |title=Elements of Information Theory |date=18 July 2006 |publisher=Wiley |location=Hoboken, New Jersey |isbn=0-471-24195-4}}</ref>{{rp|16}}

{{Equation box 1
|indent =
|title=
|equation = {{NumBlk||<math>\Eta(X,Y) = -\sum_{x\in\mathcal X} \sum_{y\in\mathcal Y} P(x,y) \log_2[P(x,y)]</math>|{{EquationRef|Eq.1}}}}
|cellpadding= 6
|border
|border colour = #0073CF
|background colour=#F5FFFA}}

where <math>x</math> and <math>y</math> are particular values of <math>X</math> and <math>Y</math>, respectively, <math>P(x,y)</math> is the joint probability of these values occurring together, and <math>P(x,y) \log_2[P(x,y)]</math> is defined to be 0 if <math>P(x,y)=0</math>.

For more than two random variables <math>X_1, ..., X_n</math> this expands to

{{Equation box 1
|indent =
|title=
|equation = {{NumBlk||<math>\Eta(X_1, ..., X_n) = 
-\sum_{x_1 \in\mathcal X_1} ... \sum_{x_n \in\mathcal X_n} P(x_1, ..., x_n) \log_2[P(x_1, ..., x_n)]</math>|{{EquationRef|Eq.2}}}}
|cellpadding= 6
|border
|border colour = #0073CF
|background colour=#F5FFFA}}

where <math>x_1,...,x_n</math> are particular values of <math>X_1,...,X_n</math>, respectively, <math>P(x_1, ..., x_n)</math> is the probability of these values occurring together, and <math>P(x_1, ..., x_n) \log_2[P(x_1, ..., x_n)]</math> is defined to be 0 if <math>P(x_1, ..., x_n)=0</math>.

==Properties==

===Nonnegativity===

The joint entropy of a set of random variables is a nonnegative number.

:<math>\Eta(X,Y) \geq 0</math>

:<math>\Eta(X_1,\ldots, X_n) \geq 0</math>

===Greater than individual entropies===

The joint entropy of a set of variables is greater than or equal to the maximum of all of the individual entropies of the variables in the set.

:<math>\Eta(X,Y) \geq \max \left[\Eta(X),\Eta(Y) \right]</math>

:<math>\Eta \bigl(X_1,\ldots, X_n \bigr) \geq \max_{1 \le i \le n} 
    \Bigl\{ \Eta\bigl(X_i\bigr) \Bigr\}</math>

===Less than or equal to the sum of individual entropies===

The joint entropy of a set of variables is less than or equal to the sum of the individual entropies of the variables in the set.  This is an example of [[Subadditivity|subadditivity]].  This inequality is an equality if and only if <math>X</math> and <math>Y</math> are statistically independent.<ref name=cover1991 />{{rp|30}}

:<math>\Eta(X,Y) \leq \Eta(X) + \Eta(Y)</math>

:<math>\Eta(X_1,\ldots, X_n) \leq \Eta(X_1) + \ldots + \Eta(X_n)</math>

==Relations to other entropy measures==

Joint entropy is used in the definition of [[Conditional entropy|conditional entropy]]<ref name=cover1991 />{{rp|22}}

:<math>\Eta(X|Y) = \Eta(X,Y) - \Eta(Y)\,</math>,

and <math display="block">\Eta(X_1,\dots,X_n) = \sum_{k=1}^n \Eta(X_k|X_{k-1},\dots, X_1)</math>It is also used in the definition of [[Mutual information|mutual information]]<ref name=cover1991 />{{rp|21}}

:<math>\operatorname{I}(X;Y) = \Eta(X) + \Eta(Y) - \Eta(X,Y)\,</math>

In quantum information theory, the joint entropy is generalized into the [[Joint quantum entropy|joint quantum entropy]].

==Joint differential entropy==
===Definition===
The above definition is for discrete random variables and just as valid in the case of continuous random variables. The continuous version of discrete joint entropy is called ''joint differential (or continuous) entropy''. Let <math>X</math> and <math>Y</math> be a continuous random variables with a joint probability density function <math>f(x,y)</math>. The differential joint entropy <math>h(X,Y)</math> is defined as<ref name=cover1991 />{{rp|249}}

{{Equation box 1
|indent =
|title=
|equation = {{NumBlk||<math>h(X,Y) = -\int_{\mathcal X , \mathcal Y} f(x,y)\log f(x,y)\,dx dy</math>|{{EquationRef|Eq.3}}}}
|cellpadding= 6
|border
|border colour = #0073CF
|background colour=#F5FFFA}}

For more than two continuous random variables <math>X_1, ..., X_n</math> the definition is generalized to:

{{Equation box 1
|indent =
|title=
|equation = {{NumBlk||<math>h(X_1, \ldots,X_n) = -\int f(x_1, \ldots,x_n)\log f(x_1, \ldots,x_n)\,dx_1 \ldots dx_n</math>|{{EquationRef|Eq.4}}}}
|cellpadding= 6
|border
|border colour = #0073CF
|background colour=#F5FFFA}}

The [[Integral|integral]] is taken over the support of <math>f</math>. It is possible that the integral does not exist in which case we say that the differential entropy is not defined.

===Properties===
As in the discrete case the joint differential entropy of a set of random variables is smaller or equal than the sum of the entropies of the individual random variables:
:<math>h(X_1,X_2, \ldots,X_n) \le \sum_{i=1}^n h(X_i)</math><ref name=cover1991 />{{rp|253}}

The following chain rule holds for two random variables:
:<math>h(X,Y) = h(X|Y) + h(Y)</math>
In the case of more than two random variables this generalizes to:<ref name=cover1991 />{{rp|253}}
:<math>h(X_1,X_2, \ldots,X_n) = \sum_{i=1}^n h(X_i|X_1,X_2, \ldots,X_{i-1})</math>
Joint differential entropy is also used in the definition of the [[Mutual information|mutual information]] between continuous random variables:
:<math>\operatorname{I}(X,Y)=h(X)+h(Y)-h(X,Y)</math>

== References ==
{{Reflist}}

[[Category:Entropy and information]]

de:Bedingte Entropie#Blockentropie

{{Sourceattribution|Joint entropy}}