Law of the unconscious statistician

Short description: Theorem in probability and statistics

In probability theory and statistics, the law of the unconscious statistician, or LOTUS, is a theorem which expresses the expected value of a function g(X) of a random variable X in terms of g and the probability distribution of X.

The form of the law depends on the type of random variable X in question. If the distribution of X is discrete and one knows its probability mass function p_X, then the expected value of g(X) is

[math]\displaystyle{ \operatorname{E}[g(X)] = \sum_x g(x) p_X(x), \, }[/math]

where the sum is over all possible values x of X. If instead the distribution of X is continuous with probability density function f_X, then the expected value of g(X) is

[math]\displaystyle{ \operatorname{E}[g(X)] = \int_{-\infty}^\infty g(x) f_X(x) \, \mathrm{d}x }[/math]

Both of these special cases can be expressed in terms of the cumulative probability distribution function F_X of X, with the expected value of g(X) now given by the Lebesgue–Stieltjes integral

[math]\displaystyle{ \operatorname{E}[g(X)] = \int_{-\infty}^\infty g(x) \, \mathrm{d}F_X(x). }[/math]

In even greater generality, X could be a random element in any measurable space, in which case the law is given in terms of measure theory and the Lebesgue integral. In this setting, there is no need to restrict the context to probability measures, and the law becomes a general theorem of mathematical analysis on Lebesgue integration relative to a pushforward measure.

Etymology

This proposition is (sometimes) known as the law of the unconscious statistician because of a purported tendency to think of the identity as the very definition of the expected value, rather than (more formally) as a consequence of its true definition.({{{1}}}, {{{2}}}) The naming is sometimes attributed to Sheldon Ross' textbook Introduction to Probability Models, although he removed the reference in later editions.({{{1}}}, {{{2}}}) Many statistics textbooks do present the result as the definition of expected value.({{{1}}}, {{{2}}})

Joint distributions

A similar property holds for joint distributions, or equivalently, for random vectors. For discrete random variables X and Y, a function of two variables g, and joint probability mass function [math]\displaystyle{ p_{X, Y}(x, y) }[/math]:({{{1}}}, {{{2}}})

[math]\displaystyle{ \operatorname{E}[g(X, Y)] = \sum_y \sum_x g(x, y) p_{X, Y}(x, y) }[/math]

In the absolutely continuous case, with [math]\displaystyle{ f_{X, Y}(x, y) }[/math] being the joint probability density function,

[math]\displaystyle{ \operatorname{E}[g(X, Y)] = \int_{-\infty}^\infty \int_{-\infty}^\infty g(x, y) f_{X, Y}(x, y) \, \mathrm{d}x \, \mathrm{d}y }[/math]

Special cases

A number of special cases are given here. In the simplest case, where the random variable X takes on countably many values (so that its distribution is discrete), the proof is particularly simple, and holds without modification if X is a discrete random vector or even a discrete random element.

The case of a continuous random variable is more subtle, since the proof in generality requires subtle forms of the change-of-variables formula for integration. However, in the framework of measure theory, the discrete case generalizes straightforwardly to general (not necessarily discrete) random elements, and the case of a continuous random variable is then a special case by making use of the Radon–Nikodym theorem.

Discrete case

Suppose that X is a random variable which takes on only finitely or countably many different values x₁, x₂, ..., with probabilities p₁, p₂, .... Then for any function g of these values, the random variable g(X) has values g(x₁), g(x₂), ..., although some of these may coincide with each other. For example, this is the case if X can take on both values 1 and −1 and g(x) = x².

Let y₁, y₂, ... enumerate the possible distinct values of [math]\displaystyle{ g(X) }[/math], and for each i let I_i denote the collection of all j with g(x_j) = y_i. Then, according to the definition of expected value, there is

[math]\displaystyle{ \operatorname{E}[g(X)]=\sum_i y_i p_{g(X)}(y_i). }[/math]

Since a [math]\displaystyle{ y_i }[/math] can be the image of multiple, distinct [math]\displaystyle{ x_j }[/math], it holds that

[math]\displaystyle{ p_{g(X)}(y_i) = \sum_{j \in I_i} p_X(x_j). }[/math]

Then the expected value can be rewritten as

[math]\displaystyle{ \sum_i y_i p_{g(X)}(y_i) = \sum_i y_i \sum_{j \in I_i} p_X(x_j) = \sum_i \sum_{j \in I_i} g(x_j) p_X(x_j) = \sum_x g(x)p_X(x). }[/math]

This equality relates the average of the outputs of g(X) as weighted by the probabilities of the outputs themselves to the average of the outputs of g(X) as weighted by the probabilities of the outputs of X.

If X takes on only finitely many possible values, the above is fully rigorous. However, if X takes on countably many values, the last equality given does not always hold, as seen by the Riemann series theorem. Because of this, it is necessary to assume the absolute convergence of the sums in question.({{{1}}}, {{{2}}})

Continuous case

Suppose that X is a random variable whose distribution has a continuous density f. If g is a general function, then the probability that g(X) is valued in a set of real numbers K equals the probability that X is valued in g⁻¹(K), which is given by

[math]\displaystyle{ \int_{g^{-1}(K)} f(x)\,\mathrm{d}x. }[/math]

Under various conditions on g, the change-of-variables formula for integration can be applied to relate this to an integral over K, and hence to identify the density of g(X) in terms of the density of X. In the simplest case, if g is differentiable with nowhere-vanishing derivative, then the above integral can be written as

[math]\displaystyle{ \int_K f(g^{-1}(y))(g^{-1})'(y)\,\mathrm{d}y, }[/math]

thereby identifying g(X) as possessing the density f (g⁻¹(y))(g⁻¹)′(y). The expected value of g(X) is then identified as

[math]\displaystyle{ \int_{-\infty}^\infty yf(g^{-1}(y))(g^{-1})'(y)\,\mathrm{d}y=\int_{-\infty}^\infty g(x)f(x)\,\mathrm{d}x, }[/math]

where the equality follows by another use of the change-of-variables formula for integration. This shows that the expected value of g(X) is encoded entirely by the function g and the density f of X.({{{1}}}, {{{2}}})

The assumption that g is differentiable with nonvanishing derivative, which is necessary for applying the usual change-of-variables formula, excludes many typical cases, such as g(x) = x². The result still holds true in these broader settings, although the proof requires more sophisticated results from mathematical analysis such as Sard's theorem and the coarea formula. In even greater generality, using the Lebesgue theory as below, it can be found that the identity

[math]\displaystyle{ \operatorname{E}[g(X)]=\int_{-\infty}^\infty g(x)f(x)\,\mathrm{d}x }[/math]

holds true whenever X has a density f (which does not have to be continuous) and whenever g is a measurable function for which g(X) has finite expected value. (Every continuous function is measurable.) Furthermore, without modification to the proof, this holds even if X is a random vector (with density) and g is a multivariable function; the integral is then taken over the multi-dimensional range of values of X.

Measure-theoretic formulation

An abstract and general form of the result is available using the framework of measure theory and the Lebesgue integral. Here, the setting is that of a measure space (Ω, μ) and a measurable map X from Ω to a measurable space Ω'. The theorem then says that for any measurable function g on Ω' which is valued in real numbers (or even the extended real number line), there is

[math]\displaystyle{ \int_\Omega g \circ X \, \mathrm{d}\mu = \int_{\Omega'} g \, \mathrm{d}(X_\sharp \mu), }[/math]

(interpreted as saying, in particular, that either side of the equality exists if the other side exists). Here X_♯ μ denotes the pushforward measure on Ω'. The 'discrete case' given above is the special case arising when X takes on only countably many values and μ is a probability measure. In fact, the discrete case (although without the restriction to probability measures) is the first step in proving the general measure-theoretic formulation, as the general version follows therefrom by an application of the monotone convergence theorem.({{{1}}}, {{{2}}}) Without any major changes, the result can also be formulated in the setting of outer measures.({{{1}}}, {{{2}}})

If μ is a σ-finite measure, the theory of the Radon–Nikodym derivative is applicable. In the special case that the measure X_♯ μ is absolutely continuous relative to some background σ-finite measure ν on Ω', there is a real-valued function f_X on Ω' representing the Radon–Nikodym derivative of the two measures, and then

[math]\displaystyle{ \int_{\Omega'} g \, \mathrm{d}(X_\sharp \mu)=\int_{\Omega'}gf_X\,\mathrm{d}\nu. }[/math]

In the further special case that Ω' is the real number line, as in the contexts discussed above, it is natural to take ν to be the Lebesgue measure, and this then recovers the 'continuous case' given above whenever μ is a probability measure. (In this special case, the condition of σ-finiteness is vacuous, since Lebesgue measure and every probability measure are trivially σ-finite.)({{{1}}}, {{{2}}})

References

• Bogachev, V. I. (2007). Measure theory. Volume I. Berlin: Springer-Verlag. doi:10.1007/978-3-540-34514-5. ISBN 978-3-540-34513-8. 
• Casella, George; Berger, Roger L. (2001). Statistical inference. Duxbury Advanced Series (Second edition of 1990 original ed.). Pacific Grove, CA: Duxbury. ISBN 0-534-11958-1. 
• Cohn, Donald L. (2013). Measure theory. Birkhäuser Advanced Texts: Basler Lehrbücher (Second edition of 1980 original ed.). New York: Birkhäuser/Springer. doi:10.1007/978-1-4614-6956-8. ISBN 978-1-4614-6955-1. 
• DeGroot, Morris H.; Schervish, Mark J. (2014). Probability and statistics (Fourth edition of 1975 original ed.). Pearson Education. ISBN 0-321-50046-6. 
• Federer, Herbert (1969). Geometric measure theory. Die Grundlehren der mathematischen Wissenschaften. 153. Berlin–Heidelberg–New York: Springer-Verlag. doi:10.1007/978-3-642-62010-2. ISBN 978-3-540-60656-7. 
• Feller, William (1968). An introduction to probability theory and its applications. Volume I (Third edition of 1950 original ed.). New York–London–Sydney: John Wiley & Sons, Inc.. 
• Halmos, Paul R. (1950). Measure theory. New York: D. Van Nostrand Co., Inc.. doi:10.1007/978-1-4684-9440-2. 
• Papoulis, Athanasios; Pillai, S. Unnikrishna (2002). Probability, random variables, and stochastic processes (Fourth edition of 1965 original ed.). New York: McGraw-Hill. ISBN 0-07-366011-6. 
• Ross, Sheldon M. (2019). Introduction to probability models (Twelfth edition of 1972 original ed.). London: Academic Press. doi:10.1016/C2017-0-01324-1. ISBN 978-0-12-814346-9.