Law of total variance

Short description: Theorem in probability theory

The law of total variance is a fundamental result in probability theory that expresses the variance of a random variable Y in terms of its conditional variances and conditional means given another random variable X. Informally, it states that the overall variability of Y can be split into an “unexplained” component (the average of within-group variances) and an “explained” component (the variance of group means).

Formally, if X and Y are random variables on the same probability space, and Y has finite variance, then:

$$\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}(Y\mid X)]+\mathrm{Var}(\mathrm{E}[Y\mid X]).$$

This identity is also known as the variance decomposition formula, the conditional variance formula, the law of iterated variances, or colloquially as Eve’s law, in parallel to the “Adam’s law” naming for the law of total expectation.

In actuarial science (particularly in credibility theory), the two terms $\mathrm{E}[\mathrm{Var}(Y\mid X)]$ and $\mathrm{Var}(\mathrm{E}[Y\mid X])$ are called the expected value of the process variance (EVPV) and the variance of the hypothetical means (VHM) respectively.

Explanation

Let Y be a random variable and X another random variable on the same probability space. The law of total variance can be understood by noting:

1. $\mathrm{Var}(Y\mid X)$ measures how much Y varies around its conditional mean $\mathrm{E}[Y\mid X].$
2. Taking the expectation of this conditional variance across all values of X gives $\mathrm{E}[\mathrm{Var}(Y\mid X)]$, often termed the “unexplained” or within-group part.
3. The variance of the conditional mean, $\mathrm{Var}(\mathrm{E}[Y\mid X])$, measures how much these conditional means differ (i.e. the “explained” or between-group part).

Adding these components yields the total variance $\mathrm{Var}(Y)$, mirroring how analysis of variance partitions variation.

Examples

Example 1 (Exam Scores)

Suppose five students take an exam scored 0–100. Let Y = student’s score and X indicate whether the student is *international* or *domestic*:

Student	Y (Score)	X
1	20	International
2	30	International
3	100	International
4	40	Domestic
5	60	Domestic

• Mean and variance for international: $\mathrm{E}[Y\mid X=\text{Intl}]=50,\mathrm{Var}(Y\mid X=\text{Intl})\approx 1266.7.$
• Mean and variance for domestic: $\mathrm{E}[Y\mid X=\text{Dom}]=50,\mathrm{Var}(Y\mid X=\text{Dom})=100.$

Both groups share the same mean (50), so the explained variance $\mathrm{Var}(\mathrm{E}[Y\mid X])$ is 0, and the total variance equals the average of the within-group variances (weighted by group size), i.e. 800.

Example 2 (Mixture of Two Gaussians)

Let X be a coin flip taking values Heads with probability h and Tails with probability 1−h. Given Heads, $Y\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}({\mu }_h,{\sigma }_h^2)$; given Tails, $Y\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}({\mu }_t,{\sigma }_t^2)$. Then $\mathrm{E}[\mathrm{Var}(Y\mid X)]=h{\sigma }_h^2+(1-h){\sigma }_t^2,$ $\mathrm{Var}(\mathrm{E}[Y\mid X])=h(1-h)({\mu }_h-{\mu }_t{)}^2,$ so $\mathrm{Var}(Y)=h{\sigma }_h^2+(1-h){\sigma }_t^2+h(1-h)({\mu }_h-{\mu }_t{)}^2.$

Example 3 (Dice and Coins)

Consider a two-stage experiment:

1. Roll a fair die (values 1–6) to choose one of six biased coins.
2. Flip that chosen coin; let Y=1 if Heads, 0 if Tails.

Then $\mathrm{E}[Y\mid X=i]=p_i,\mathrm{Var}(Y\mid X=i)=p_i(1-p_i).$ The overall variance of Y becomes $$\mathrm{Var}(Y)=\mathrm{E}[p_X(1-p_X)]+\mathrm{Var}\left(p_X\right),$$ with $p_X$ uniform on $\{p_1,\dots ,p_6\}.$

Proof

Discrete/Finite Proof

Let $(X_i,Y_i)$, $i=1,\dots ,n$, be observed pairs. Define $\bar{Y}=\mathrm{E}[Y].$ Then $$\begin{array}{rl}\mathrm{Var}(Y)& =\frac{1}{n}\sum _{i=1}^n{(Y_i-\bar{Y})}^2\\ & =\frac{1}{n}\sum _{i=1}^n{[(Y_i-{\bar{Y}}_{X_i})+({\bar{Y}}_{X_i}-\bar{Y})]}^2,\end{array}$$ where ${\bar{Y}}_{X_i}=\mathrm{E}[Y\mid X=X_i].$ Expanding the square and noting the cross term cancels in summation yields: $\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}(Y\mid X)]+\mathrm{Var}(\mathrm{E}[Y\mid X]).$

General Case

Using $\mathrm{Var}(Y)=\mathrm{E}[Y^2]-\mathrm{E}[Y{]}^2$ and the law of total expectation: $$\begin{array}{rl}\mathrm{E}[Y^2]& =\mathrm{E}[\mathrm{E}(Y^2\mid X)]\\ & =\mathrm{E}[\mathrm{Var}(Y\mid X)+\mathrm{E}[Y\mid X{]}^2].\end{array}$$ Subtract $\mathrm{E}[Y{]}^2={(\mathrm{E}[\mathrm{E}(Y\mid X)])}^2$ and regroup to arrive at $\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}(Y\mid X)]+\mathrm{Var}(\mathrm{E}[Y\mid X]).$

Applications

Analysis of Variance (ANOVA)

In a one-way analysis of variance, the total sum of squares (proportional to $\mathrm{Var}(Y)$) is split into a “between-group” sum of squares ($\mathrm{Var}(\mathrm{E}[Y\mid X])$) plus a “within-group” sum of squares ($\mathrm{E}[\mathrm{Var}(Y\mid X)]$). The F-test examines whether the explained component is sufficiently large to indicate X has a significant effect on Y.

Regression and R²

In linear regression and related models, if $\hat{Y}=\mathrm{E}[Y\mid X],$ the fraction of variance explained is $$\begin{array}{rl}{R}^2=\frac{\mathrm{Var}(\hat{Y})}{\mathrm{Var}(Y)}& =\frac{\mathrm{Var}(\mathrm{E}[Y\mid X])}{\mathrm{Var}(Y)}\\ & =1-\frac{\mathrm{E}[\mathrm{Var}(Y\mid X)]}{\mathrm{Var}(Y)}.\end{array}$$ In the simple linear case (one predictor), $R^2$ also equals the square of the Pearson correlation coefficient between X and Y.

Machine Learning and Bayesian Inference

In many Bayesian and ensemble methods, one decomposes prediction uncertainty via the law of total variance. For a Bayesian neural network with random parameters $\theta$: $\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}(Y\mid \theta )]+\mathrm{Var}(\mathrm{E}[Y\mid \theta ]),$ often referred to as “aleatoric” (within-model) vs. “epistemic” (between-model) uncertainty.

Actuarial Science

Credibility theory uses the same partitioning: the expected value of process variance (EVPV), $\mathrm{E}[\mathrm{Var}(Y\mid X)],$ and the variance of hypothetical means (VHM), $\mathrm{Var}(\mathrm{E}[Y\mid X]).$ The ratio of explained to total variance determines how much “credibility” to give to individual risk classifications.

Information Theory

For jointly Gaussian $(X,Y)$, the fraction $\mathrm{Var}(\mathrm{E}[Y\mid X])/\mathrm{Var}(Y)$ relates directly to the mutual information $I(Y;X).$ In non-Gaussian settings, a high explained-variance ratio still indicates significant information about Y contained in X.

Generalizations

The law of total variance generalizes to multiple or nested conditionings. For example, with two conditioning variables $X_1$ and $X_2$: $$\mathrm{Var}(Y)=\mathrm{E}[\mathrm{Var}(Y\mid X_1,X_2)]+\mathrm{E}[\mathrm{Var}(\mathrm{E}[Y\mid X_1,X_2]\mid X_1)]+\mathrm{Var}(\mathrm{E}[Y\mid X_1]).$$ More generally, the law of total cumulance extends this approach to higher moments.

See also

• Law of total expectation (Adam’s law)
• Law of total covariance
• Law of total cumulance
• Analysis of variance
• Conditional expectation
• R-squared
• Fraction of variance unexplained
• Variance decomposition

References

• Blitzstein, Joe. "Stat 110 Final Review (Eve's Law)". stat110.net. Harvard University, Department of Statistics. http://projects.iq.harvard.edu/files/stat110/files/final_review.pdf. 
• "Law of total variance". https://statproofbook.github.io/P/var-tot.html. 
• Billingsley, Patrick (1995). "Problem 34.10(b)". Probability and Measure. New York, NY: John Wiley & Sons, Inc.. ISBN 0-471-00710-2. 
• Weiss, Neil A. (2005). A Course in Probability. Addison–Wesley. pp. 380–386. ISBN 0-201-77471-2. 
• Bowsher, C.G.; Swain, P.S. (2012). "Identifying sources of variation and the flow of information in biochemical networks". PNAS 109 (20): E1320–E1328. doi:10.1073/pnas.1118365109.