Cross entropy
Information theory 

In information theory, the crossentropy between two probability distributions [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math] over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution [math]\displaystyle{ q }[/math], rather than the true distribution [math]\displaystyle{ p }[/math].
Definition
The crossentropy of the distribution [math]\displaystyle{ q }[/math] relative to a distribution [math]\displaystyle{ p }[/math] over a given set is defined as follows:
 [math]\displaystyle{ H(p, q) = \operatorname{E}_p[\log q] }[/math],
where [math]\displaystyle{ E_p[\cdot] }[/math] is the expected value operator with respect to the distribution [math]\displaystyle{ p }[/math].
The definition may be formulated using the Kullback–Leibler divergence [math]\displaystyle{ D_{\mathrm{KL}}(p \parallel q) }[/math], divergence of [math]\displaystyle{ p }[/math] from [math]\displaystyle{ q }[/math] (also known as the relative entropy of [math]\displaystyle{ p }[/math] with respect to [math]\displaystyle{ q }[/math]).
 [math]\displaystyle{ H(p, q) = H(p) + D_{\mathrm{KL}}(p \parallel q), }[/math]
where [math]\displaystyle{ H(p) }[/math] is the entropy of [math]\displaystyle{ p }[/math].
For discrete probability distributions [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math] with the same support [math]\displaystyle{ \mathcal{X} }[/math] this means
[math]\displaystyle{ H(P,Q) = \sum_{x\in\mathcal{X}} p(x)\, \log q(x) }[/math] 

( ) 
The situation for continuous distributions is analogous. We have to assume that [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math] are absolutely continuous with respect to some reference measure [math]\displaystyle{ r }[/math] (usually [math]\displaystyle{ r }[/math] is a Lebesgue measure on a Borel σalgebra). Let [math]\displaystyle{ P }[/math] and [math]\displaystyle{ Q }[/math] be probability density functions of [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math] with respect to [math]\displaystyle{ r }[/math]. Then
 [math]\displaystyle{ \int_\mathcal{X} P(x)\, \log Q(x)\, dr(x) = \operatorname{E}_p[\log Q] }[/math]
and therefore
[math]\displaystyle{ H(p,q) = \int_\mathcal{X} P(x)\, \log Q(x)\, dr(x) }[/math] 

( ) 
NB: The notation [math]\displaystyle{ H(p,q) }[/math] is also used for a different concept, the joint entropy of [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math].
Motivation
In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value [math]\displaystyle{ x_i }[/math] out of a set of possibilities [math]\displaystyle{ \{x_1,\ldots,x_n\} }[/math] can be seen as representing an implicit probability distribution [math]\displaystyle{ q(x_i) = \left(\frac{1}{2}\right)^{\ell_i} }[/math] over [math]\displaystyle{ \{x_1, \ldots, x_n\} }[/math], where [math]\displaystyle{ \ell_i }[/math] is the length of the code for [math]\displaystyle{ x_i }[/math] in bits. Therefore, crossentropy can be interpreted as the expected messagelength per datum when a wrong distribution [math]\displaystyle{ q }[/math] is assumed while the data actually follows a distribution [math]\displaystyle{ p }[/math]. That is why the expectation is taken over the true probability distribution [math]\displaystyle{ p }[/math] and not [math]\displaystyle{ q }[/math]. Indeed the expected messagelength under the true distribution [math]\displaystyle{ p }[/math] is
 [math]\displaystyle{ \operatorname{E}_p[\ell] =  \operatorname{E}_p\left[\frac{\ln{q(x)}}{\ln(2)}\right] =  \operatorname{E}_p\left[\log_2 {q(x)}\right] =  \sum_{x_i} p(x_i)\, \log_2 q(x_i) = \sum_x p(x)\, \log_2 q(x) = H(p, q). }[/math]
Estimation
There are many situations where crossentropy needs to be measured but the distribution of [math]\displaystyle{ p }[/math] is unknown. An example is language modeling, where a model is created based on a training set [math]\displaystyle{ T }[/math], and then its crossentropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, [math]\displaystyle{ p }[/math] is the true distribution of words in any corpus, and [math]\displaystyle{ q }[/math] is the distribution of words as predicted by the model. Since the true distribution is unknown, crossentropy cannot be directly calculated. In these cases, an estimate of crossentropy is calculated using the following formula:
 [math]\displaystyle{ H(T,q) = \sum_{i=1}^N \frac{1}{N} \log_2 q(x_i) }[/math]
where [math]\displaystyle{ N }[/math] is the size of the test set, and [math]\displaystyle{ q(x) }[/math] is the probability of event [math]\displaystyle{ x }[/math] estimated from the training set. In other words, [math]\displaystyle{ q(x_i) }[/math] is the probability estimate of the model that the ith word of the text is [math]\displaystyle{ x_i }[/math]. The sum is averaged over the [math]\displaystyle{ N }[/math] words of the test. This is a Monte Carlo estimate of the true crossentropy, where the test set is treated as samples from [math]\displaystyle{ p(x) }[/math]^{[citation needed]}.
Relation to maximum likelihood
In classification problems we want to estimate the probability of different outcomes. Let the estimated probability of outcome [math]\displaystyle{ i }[/math] be [math]\displaystyle{ q_{\theta}(X=i) }[/math] with tobeoptimized parameters [math]\displaystyle{ \theta }[/math] and let the frequency (empirical probability) of outcome [math]\displaystyle{ i }[/math] in the training set be [math]\displaystyle{ p(X=i) }[/math]. Given N conditionally independent samples in the training set, then the likelihood of the parameters [math]\displaystyle{ \theta }[/math] of the model [math]\displaystyle{ q_{\theta}(X=x) }[/math] on the training set is
 [math]\displaystyle{ \mathcal{L}(\theta)=\prod_{i \in X} (\mbox{est. probability of } i)^{\mbox{number of occurrences of } i} = \prod_i q_{\theta}(X=i)^{N p(X=i)} }[/math]
where the last expression is due to the definition of the multinomial PMF. Therefore, the loglikelihood, divided by [math]\displaystyle{ N }[/math] is
 [math]\displaystyle{ \frac{1}{N}\log(\mathcal{L}(\theta))=\frac{1}{N} \log \prod_i q_{\theta}(X=i)^{N p(X=i)} = \sum_i p(X=i) \log q_{\theta}(X=i) = H(p, q) }[/math]
so that maximizing the likelihood with respect to the parameters [math]\displaystyle{ \theta }[/math] is the same as minimizing the crossentropy.^{[citation needed]}
Crossentropy minimization
Crossentropy minimization is frequently used in optimization and rareevent probability estimation. When comparing a distribution [math]\displaystyle{ q }[/math] against a fixed reference distribution [math]\displaystyle{ p }[/math], crossentropy and KL divergence are identical up to an additive constant (since [math]\displaystyle{ p }[/math] is fixed): According to the Gibbs' inequality, both take on their minimal values when [math]\displaystyle{ p = q }[/math], which is [math]\displaystyle{ 0 }[/math] for KL divergence, and [math]\displaystyle{ \mathrm{H}(p) }[/math] for crossentropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum CrossEntropy (MCE), or Minxent.
However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution [math]\displaystyle{ q }[/math] is the fixed prior reference distribution, and the distribution [math]\displaystyle{ p }[/math] is optimized to be as close to [math]\displaystyle{ q }[/math] as possible, subject to some constraint. In this case the two minimizations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by restating crossentropy to be [math]\displaystyle{ D_{\mathrm{KL}}(p \parallel q) }[/math], rather than [math]\displaystyle{ H(p, q) }[/math]. In fact, crossentropy is another name for relative entropy, see Cover and Thomas ^{[1]} and Good.^{[2]} On the other hand, [math]\displaystyle{ H(p, q) }[/math] does not agree with the literature and can be misleading.
Crossentropy loss function and logistic regression
Crossentropy can be used to define a loss function in machine learning and optimization. The true probability [math]\displaystyle{ p_i }[/math] is the true label, and the given distribution [math]\displaystyle{ q_i }[/math] is the predicted value of the current model. This is also known as the log loss (or logarithmic loss^{[3]} or logistic loss);^{[4]} the terms "log loss" and "crossentropy loss" are used interchangeably.^{[5]}
More specifically, consider a binary regression model which can be used to classify observations into two possible classes (often simply labelled [math]\displaystyle{ 0 }[/math] and [math]\displaystyle{ 1 }[/math]). The output of the model for a given observation, given a vector of input features [math]\displaystyle{ x }[/math], can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function [math]\displaystyle{ g(z) = 1/(1+e^{z}) }[/math] where [math]\displaystyle{ z }[/math] is some function of the input vector [math]\displaystyle{ x }[/math], commonly just a linear function. The probability of the output [math]\displaystyle{ y=1 }[/math] is given by
 [math]\displaystyle{ q_{y=1} = \hat{y} \equiv g(\mathbf{w}\cdot\mathbf{x}) = \frac 1 {1+e^{\mathbf{w}\cdot\mathbf{x}}}, }[/math]
where the vector of weights [math]\displaystyle{ \mathbf{w} }[/math] is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output [math]\displaystyle{ y=0 }[/math] is simply given by
 [math]\displaystyle{ q_{y=0} = 1\hat{y} }[/math]
Having set up our notation, [math]\displaystyle{ p\in\{y,1y\} }[/math] and [math]\displaystyle{ q\in\{\hat{y},1\hat{y}\} }[/math], we can use crossentropy to get a measure of dissimilarity between [math]\displaystyle{ p }[/math] and [math]\displaystyle{ q }[/math]:
 [math]\displaystyle{ H(p,q)\ =\ \sum_i p_i\log q_i\ =\ y\log\hat{y}  (1y)\log(1\hat{y}) }[/math]
Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average crossentropy in the sample. For example, suppose we have [math]\displaystyle{ N }[/math] samples with each sample indexed by [math]\displaystyle{ n=1,\dots,N }[/math]. The average of the loss function is then given by:
 [math]\displaystyle{ \begin{align} J(\mathbf{w})\ &=\ \frac1N\sum_{n=1}^N H(p_n,q_n)\ =\ \frac1N\sum_{n=1}^N\ \bigg[y_n \log \hat y_n + (1  y_n) \log (1  \hat y_n)\bigg]\,, \end{align} }[/math]
where [math]\displaystyle{ \hat{y}_n\equiv g(\mathbf{w}\cdot\mathbf{x}_n) = 1/(1+e^{\mathbf{w}\cdot\mathbf{x}_n}) }[/math], with [math]\displaystyle{ g(z) }[/math] the logistic function as before.
The logistic loss is sometimes called crossentropy loss. It is also known as log loss (In this case, the binary label is often denoted by {−1,+1}).^{[6]}
Remark: The gradient of the crossentropy loss for logistic regression is the same as the gradient of the squared error loss for linear regression. That is, define
 [math]\displaystyle{ X^T=\begin{pmatrix} 1&x_{11}&\dots&x_{1p}\\ 1&x_{21}&\cdots&x_{2p}\\ \vdots & \vdots && \vdots \\ 1&x_{n1}&\cdots&x_{np}\\ \end{pmatrix}\in \mathbb{R}^{n\times(p+1)} }[/math]
 [math]\displaystyle{ \hat{y_i} = \hat{f}(x_{i1},\dots,x_{ip}) = \frac{1}{1+\exp(\beta_0\beta_1x_{i1}\dots\beta_px_{ip})} }[/math]
 [math]\displaystyle{ L(\overrightarrow{\beta})=\sum_{i=1}^N [y_i\log \hat{y}_i+(1y_i)\log(1\hat{y}_i)] }[/math]
Then we have the result
 [math]\displaystyle{ \frac{\partial}{\partial\overrightarrow{\beta}}L(\overrightarrow{\beta})=X^T(\hat{Y}Y) }[/math]
The proof is as follows. For any [math]\displaystyle{ \hat{y}_i }[/math], we have
 [math]\displaystyle{ \frac{\partial}{\partial\beta_0}\ln\frac{1}{1+e^{\beta_0+k_0}} = \frac{e^{\beta_0+k_0}}{1+e^{\beta_0+k_0}} }[/math]
 [math]\displaystyle{ \frac{\partial}{\partial \beta_0}\ln \left(1\frac{1}{1+e^{\beta_0+k_0}}\right)=\frac{1}{1+e^{\beta_0+k_0}} }[/math]
 [math]\displaystyle{ \begin{align} \frac{\partial}{\partial\beta_0}L(\overrightarrow{\beta})&=\sum_{i=1}^{N}\left[\frac{y_i \cdot e^{\beta_0+k_0}}{1+e^{\beta_0+k_0}}(1y_i)\frac{1}{1+e^{\beta_0+k_0}}\right]\\ &=\sum_{i=1}^{N}[y_i\hat{y}_i] = \sum_{i=1}^{N}(\hat{y}_iy_i) \end{align} }[/math]
 [math]\displaystyle{ \frac{\partial}{\partial \beta_1}\ln \frac{1}{1+e^{\beta_1x_{i1}+k_1}} = \frac{x_{i1}e^{k_1}}{e^{\beta_1x_{i1}}+e^{k_1}} }[/math]
 [math]\displaystyle{ \frac{\partial}{\partial \beta_1}\ln\left[1\frac{1}{1+e^{\beta_1x_{i1}+k_1}}\right] = \frac{x_{i1}e^{\beta_1x_{i1}}}{e^{\beta_1x_{i1}}+e^{k_1}} }[/math]
 [math]\displaystyle{ \frac{\partial}{\partial\beta_1}L(\overrightarrow{\beta}) = \sum_{i=1}^N x_{i1}(y_i\hat{y}_i) = \sum_{i=1}^N x_{i1}(\hat{y}_iy_i) }[/math]
In a similar way, we eventually obtain the desired result.
See also
 Crossentropy method
 Logistic regression
 Conditional entropy
 Maximum likelihood estimation
 Mutual information
References
 ↑ Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd Edition, Wiley, p. 80
 ↑ I. J. Good, Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Table, Ann. of Math. Statistics, 1963
 ↑ The Mathematics of Information Coding, Extraction and Distribution, by George Cybenko, Dianne P. O'Leary, Jorma Rissanen, 1999, p. 82
 ↑ Probability for Machine Learning: Discover How To Harness Uncertainty With Python, Jason Brownlee, 2019, p. 220: "Logistic loss refers to the loss function commonly used to optimize a logistic regression model. It may also be referred to as logarithmic loss (which is confusing) or simply log loss."
 ↑ sklearn.metrics.log_loss
 ↑ Murphy, Kevin (2012). Machine Learning: A Probabilistic Perspective. MIT. ISBN 9780262018029.
External links