Law of total variance

From HandWiki
Short description: Theorem in probability theory

In probability theory, the law of total variance[1] or variance decomposition formula or conditional variance formulas or law of iterated variances also known as Eve's law,[2] states that if [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] are random variables on the same probability space, and the variance of [math]\displaystyle{ Y }[/math] is finite, then

[math]\displaystyle{ \operatorname{Var}(Y) = \operatorname{E}[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(\operatorname{E}[Y \mid X]). }[/math]

In language perhaps better known to statisticians than to probability theorists, the two terms are the "unexplained" and the "explained" components of the variance respectively (cf. fraction of variance unexplained, explained variation). In actuarial science, specifically credibility theory, the first component is called the expected value of the process variance (EVPV) and the second is called the variance of the hypothetical means (VHM).[3] These two components are also the source of the term "Eve's law", from the initials EV VE for "expectation of variance" and "variance of expectation".


Suppose X is a coin flip with the probability of heads being h. Suppose that when X = heads then Y is drawn from a normal distribution with mean μh and standard deviation σh, and that when X = tails then Y is drawn from normal distribution with mean μt and standard deviation σt. Then the first, "unexplained" term on the right-hand side of the above formula is the weighted average of the variances, h2 + (1 − h)σt2, and the second, "explained" term is the variance of the distribution that gives μh with probability h and gives μt with probability 1 − h.


There is a general variance decomposition formula for [math]\displaystyle{ c \geq 2 }[/math] components (see below).[4] For example, with two conditioning random variables: [math]\displaystyle{ \operatorname{Var}[Y] = \operatorname{E}\left[\operatorname{Var}\left(Y \mid X_1, X_2\right)\right] + \operatorname{E}[\operatorname{Var}(\operatorname{E}\left[Y \mid X_1, X_2\right] \mid X_1)] + \operatorname{Var}(\operatorname{E}\left[Y \mid X_1\right]), }[/math] which follows from the law of total conditional variance:[4] [math]\displaystyle{ \operatorname{Var}(Y \mid X_1) = \operatorname{E} \left[\operatorname{Var}(Y \mid X_1, X_2) \mid X_1\right] + \operatorname{Var} \left(\operatorname{E}\left[Y \mid X_1, X_2 \right] \mid X_1\right). }[/math]

Note that the conditional expected value [math]\displaystyle{ \operatorname{E}(Y \mid X) }[/math] is a random variable in its own right, whose value depends on the value of [math]\displaystyle{ X. }[/math] Notice that the conditional expected value of [math]\displaystyle{ Y }[/math] given the event [math]\displaystyle{ X = x }[/math] is a function of [math]\displaystyle{ x }[/math] (this is where adherence to the conventional and rigidly case-sensitive notation of probability theory becomes important!). If we write [math]\displaystyle{ \operatorname{E}(Y \mid X = x) = g(x) }[/math] then the random variable [math]\displaystyle{ \operatorname{E}(Y \mid X) }[/math] is just [math]\displaystyle{ g(X). }[/math] Similar comments apply to the conditional variance.

One special case, (similar to the law of total expectation) states that if [math]\displaystyle{ A_1, \ldots, A_n }[/math] is a partition of the whole outcome space, that is, these events are mutually exclusive and exhaustive, then [math]\displaystyle{ \begin{align} \operatorname{Var} (X) = {} & \sum_{i=1}^n \operatorname{Var}(X\mid A_i) \Pr(A_i) + \sum_{i=1}^n \operatorname{E}[X\mid A_i]^2 (1-\Pr(A_i))\Pr(A_i) \\[4pt] & {} - 2\sum_{i=2}^n \sum_{j=1}^{i-1} \operatorname{E}[X \mid A_i] \Pr(A_i)\operatorname{E}[X\mid A_j] \Pr(A_j). \end{align} }[/math]

In this formula, the first component is the expectation of the conditional variance; the other two components are the variance of the conditional expectation.


The law of total variance can be proved using the law of total expectation.[5] First, [math]\displaystyle{ \operatorname{Var}[Y] = \operatorname{E}\left[Y^2\right] - \operatorname{E}[Y]^2 }[/math] from the definition of variance. Again, from the definition of variance, and applying the law of total expectation, we have [math]\displaystyle{ \operatorname{E}\left[Y^2\right] = \operatorname{E}\left[\operatorname{E}[Y^2\mid X]\right] = \operatorname{E} \left[\operatorname{Var}[Y \mid X] + \operatorname{E}[Y \mid X]^2\right]. }[/math]

Now we rewrite the conditional second moment of [math]\displaystyle{ Y }[/math] in terms of its variance and first moment, and apply the law of total expectation on the right hand side: [math]\displaystyle{ \operatorname{E}\left[Y^2\right] - \operatorname{E}[Y]^2 = \operatorname{E} \left[\operatorname{Var}[Y \mid X] + \operatorname{E}[Y \mid X]^2\right] - \operatorname{E} [\operatorname{E}[Y \mid X]]^2. }[/math]

Since the expectation of a sum is the sum of expectations, the terms can now be regrouped: [math]\displaystyle{ = \left(\operatorname{E} [\operatorname{Var}[Y \mid X]]\right) + \left(\operatorname{E} \left[\operatorname{E}[Y \mid X]^2\right] - \operatorname{E} [\operatorname{E}[Y \mid X]]^2\right). }[/math]

Finally, we recognize the terms in the second set of parentheses as the variance of the conditional expectation [math]\displaystyle{ \operatorname{E}[Y \mid X] }[/math]: [math]\displaystyle{ = \operatorname{E} [\operatorname{Var}[Y \mid X]] + \operatorname{Var} [\operatorname{E}[Y \mid X]]. }[/math]

General variance decomposition applicable to dynamic systems

The following formula shows how to apply the general, measure theoretic variance decomposition formula [4] to stochastic dynamic systems. Let [math]\displaystyle{ Y(t) }[/math] be the value of a system variable at time [math]\displaystyle{ t. }[/math] Suppose we have the internal histories (natural filtrations) [math]\displaystyle{ H_{1t},H_{2t},\ldots,H_{c-1,t} }[/math], each one corresponding to the history (trajectory) of a different collection of system variables. The collections need not be disjoint. The variance of [math]\displaystyle{ Y(t) }[/math] can be decomposed, for all times [math]\displaystyle{ t, }[/math] into [math]\displaystyle{ c \geq 2 }[/math] components as follows: [math]\displaystyle{ \begin{align} \operatorname{Var}[Y(t)] = {} & \operatorname{E}(\operatorname{Var}[Y(t)\mid H_{1t},H_{2t},\ldots,H_{c-1,t}]) \\[4pt] & {} + \sum_{j=2}^{c-1}\operatorname{E}(\operatorname{Var}[\operatorname{E}[Y(t)\mid H_{1t},H_{2t},\ldots,H_{jt}] \mid H_{1t},H_{2t},\ldots,H_{j-1,t}]) \\[4pt] & {} + \operatorname{Var}(\operatorname{E}[Y(t)\mid H_{1t}]). \end{align} }[/math]

The decomposition is not unique. It depends on the order of the conditioning in the sequential decomposition.

The square of the correlation and explained (or informational) variation

In cases where [math]\displaystyle{ (Y, X) }[/math] are such that the conditional expected value is linear; that is, in cases where [math]\displaystyle{ \operatorname{E}(Y \mid X) = a X + b, }[/math] it follows from the bilinearity of covariance that [math]\displaystyle{ a={\operatorname{Cov}(Y, X) \over \operatorname{Var}(X)} }[/math] and [math]\displaystyle{ b = \operatorname{E}(Y)-{\operatorname{Cov}(Y, X) \over \operatorname{Var}(X)} \operatorname{E}(X) }[/math] and the explained component of the variance divided by the total variance is just the square of the correlation between [math]\displaystyle{ Y }[/math] and [math]\displaystyle{ X; }[/math] that is, in such cases, [math]\displaystyle{ {\operatorname{Var}(\operatorname{E}(Y \mid X)) \over \operatorname{Var}(Y)} = \operatorname{Corr}(X, Y)^2. }[/math]

One example of this situation is when [math]\displaystyle{ (X, Y) }[/math] have a bivariate normal (Gaussian) distribution.

More generally, when the conditional expectation [math]\displaystyle{ \operatorname{E}(Y \mid X) }[/math] is a non-linear function of [math]\displaystyle{ X }[/math][4] [math]\displaystyle{ \iota_{Y\mid X} = {\operatorname{Var}(\operatorname{E}(Y \mid X)) \over \operatorname{Var}(Y)} = \operatorname{Corr}(\operatorname{E}(Y \mid X), Y)^2, }[/math] which can be estimated as the [math]\displaystyle{ R }[/math] squared from a non-linear regression of [math]\displaystyle{ Y }[/math] on [math]\displaystyle{ X, }[/math] using data drawn from the joint distribution of [math]\displaystyle{ (X, Y). }[/math] When [math]\displaystyle{ \operatorname{E}(Y \mid X) }[/math] has a Gaussian distribution (and is an invertible function of [math]\displaystyle{ X }[/math]), or [math]\displaystyle{ Y }[/math] itself has a (marginal) Gaussian distribution, this explained component of variation sets a lower bound on the mutual information:[4] [math]\displaystyle{ \operatorname{I}(Y; X) \geq \ln \left([1 - \iota_{Y \mid X}]^{-1/2}\right). }[/math]

Higher moments

A similar law for the third central moment [math]\displaystyle{ \mu_3 }[/math] says [math]\displaystyle{ \mu_3(Y)=\operatorname{E}\left(\mu_3(Y \mid X)\right) + \mu_3(\operatorname{E}(Y \mid X)) + 3\operatorname{cov}(\operatorname{E}(Y \mid X), \operatorname{var}(Y \mid X)). }[/math]

For higher cumulants, a generalization exists. See law of total cumulance.

See also


  1. Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 385–386.
  2. Joseph K. Blitzstein and Jessica Hwang: "Introduction to Probability"
  3. Mahler, Howard C.; Dean, Curtis Gary (2001). "Chapter 8: Credibility". in Casualty Actuarial Society. Foundations of Casualty Actuarial Science (4th ed.). Casualty Actuarial Society. pp. 525–526. ISBN 978-0-96247-622-8. Retrieved June 25, 2015. 
  4. 4.0 4.1 4.2 4.3 4.4 Bowsher, C.G. and P.S. Swain, Identifying sources of variation and the flow of information in biochemical networks, PNAS May 15, 2012 109 (20) E1320-E1328.
  5. Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 380–383.