Lehmann–Scheffé theorem

Short description: Theorem in statistics

In statistics, the Lehmann–Scheffé theorem provides sufficient conditions for the existence of a best unbiased estimator in a statistical model. The theorem states that any unbiased estimator for a quantity that depends on the data only through a complete, sufficient statistic is the unique uniformly minimum-variance unbiased estimator (UMVUE) of that quantity. The Lehmann–Scheffé theorem is named after Erich Leo Lehmann and Henry Scheffé, given their two early papers.

Introduction

Given a vector $X=(X_1,X_2,\dots ,X_n)$ of random samples from a distribution ${\mathbb{\mathbb{P}}}_{\theta }$ for some parameter $\theta \in \mathrm{\Theta }$, the goal is to establish sufficient conditions for the existence of an UMVU estimator $T^{\ast }(X)$ for some quantity $g(\theta )$, that is, ${\mathbb{𝔼}}_{\theta }[T^{\ast }(X)]=g(\theta )$ and for any unbiased estimator $S(X)$ it holds

$${\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(T^{\ast }(X))\le {\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(S(X)),\mathrm{\forall }\theta \in \mathrm{\Theta }.$$

The Rao–Blackwell theorem already shows that, given a sufficient statistic $T$, the estimator $T^{\ast }(X)=\mathbb{𝔼}[S(X)\mid T(X)]$ has a uniformly smaller variance than $S(X)$, but it does not guarantee that $T^{\ast }$ is already UMVU. This is where the Lehmann–Scheffé theorem comes in, if $T$ is also complete, that is, for any real-valued measurable function $\phi$ it holds

$${\mathbb{𝔼}}_{\theta }[\phi (T(X))]=0⟹\phi (T(X))=0{\mathbb{\mathbb{P}}}_{\theta }\text{-a.s.}$$

Statement

As above, let $X=(X_1,X_2,\dots ,X_n)$ be a vector of random samples from a distribution ${\mathbb{\mathbb{P}}}_{\theta }$ for some parameter $\theta \in \mathrm{\Theta }$ and $\mathrm{\Theta }$ an arbitrary set.

Assume that there exists a complete, sufficient statistic $T$ for the family of distributions $({\mathbb{\mathbb{P}}}_{\theta }{)}_{\theta \in \mathrm{\Theta }}$. Then, the following two equivalent statements hold:

• There exists at most one measurable function $h$ such that $T^{\ast }=h(T(X))$ is unbiased for $g(\theta )$ and ${\mathbb{𝔼}}_{\theta }[T^{\ast }(X{)}^2]<\mathrm{\infty }$ for all $\theta \in \mathrm{\Theta }$, in which case $T^{\ast }(X)$ is the unique UMVUE for $g(\theta )$.
• For any unbiased estimator $S(X)$, if it exists, with ${\mathbb{𝔼}}_{\theta }[S(X{)}^2]<\mathrm{\infty }$ for all $\theta \in \mathrm{\Theta }$, the estimator $T^{\ast }(X)={\mathbb{𝔼}}_{\theta }[S(X)\mid T(X)]$ is the unique UMVUE for $g(\theta )$.

In fact, the theorem does not state that unbiased estimators exist in the first place. However, if they do, then there exists a unique square-integrable UMVUE. Moreover, the estimator $T^{\ast }(X)={\mathbb{𝔼}}_{\theta }[S(X)\mid T(X)]$ does neither depend on $\theta$, since $T$ is sufficient, nor on $S$, since $T$ is also complete.

Proof

In the following, the dependence of an estimator on the data $X$ will not be written out explicitly, i.e., we write $S$ instead of $S(X)$.

First of all, if there is no unbiased estimator for $g(\theta )$, then there is obviously no UMVUE, and if all unbiased estimator are not square-integrable, then their variances are infinity and the statement is trivial. Thus, we focus on the case where a square-integrable unbiased estimator exists.

Uniqueness of $h$: Let $S=h_1(T)$ and $R=h_2(T)$ be unbiased estimators of $g(\theta )$ for some measurable functions $h_1$ and $h_2$. For the expectation of the difference it holds$${\mathbb{𝔼}}_{\theta }[h_1(T)-h_2(T)]=g(\theta )-g(\theta )=0.$$Since $T$ is complete, this implies $h_1-h_2=0$. Thus, $T^{\ast }=h_1(T)=:h(T)$ is the unique unbiased estimator that is a function of $T$.

$h(T)$ is the UMVUE: Let $S$ be any square-integrable unbiased estimator and $S^{\ast }={\mathbb{𝔼}}_{\theta }[S\mid T]$. By the factorization lemma there exists measurable function $\tilde{h}$ such that $S^{\ast }=\tilde{h}(T)$ and since $S^{\ast }$ is unbiased as well, by the above, it must hold $S^{\ast }=\tilde{h}(T)=h(T)=T^{\ast }$. Thus, by the Rao–Blackwell theorem, it follows$${\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(T^{\ast })={\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(S^{\ast })={\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }({\mathbb{𝔼}}_{\theta }[S\mid T])\le {\mathrm{V}\mathrm{a}\mathrm{r}}_{\theta }(S),\mathrm{\forall }\theta \in \mathrm{\Theta }.$$In other words, $T^{\ast }=h(T)$ is an UMVUE and according to the first part it is unique.

Application

The Lehmann–Scheffé theorem motivates two general methods to construct UMVU estimators for $g(\theta )$ in models which allow for a complete sufficient statistic $T$.

Method 1: Determining the function $h$

The UMVUE, if it exists, is the (unique) solution of the equation$${\mathbb{𝔼}}_{\theta }[h(T)]=g(\theta )$$for all $\theta \in \mathrm{\Theta }$. If $h$ is, for example, a linear function, then solving this equation is fairly easy.

Method 2: Conditioning on an unbiased estimator

First, it suffices to find any unbiased estimator $S$ of $g(\theta )$, which is often easily feasible. The UMVUE can then be determined by evaluating the condition expectation ${\mathbb{𝔼}}_{\theta }[S\mid T]$. Since the choice of $S$ is arbitrary, it is preferable to choose it such that the conditional expectation is as simple as possible.

Examples

Bernoulli distribution

Let $X_1,\dots ,X_n$ be Bernoulli-distributed with probability $p\in (0,1)$. The joint probability mass function $f_p$ is of the form$$f_p(x_1,\dots ,x_n)=\prod _{i=1}^n{p}^{x_i}(1-p{)}^{1-x_i}={\left(\frac{p}{1-p}\right)}^{\sum _{i=1}^n{x}_i}(1-p{)}^n,x_i\in \{0,1\},$$thus, by the Fisher–Neyman factorization theorem, $T(x)=\sum _{i=1}^n{x}_i$ is a sufficient statistic. It is also complete: Let $\phi$ be any measurable function such that ${\mathbb{𝔼}}_{\theta }[\phi (T(X))]=0$. Since $T$ is $\mathrm{B}\mathrm{e}\mathrm{r}(n,p)$-distributed, this means$$0={\mathbb{𝔼}}_{\theta }[\phi (T(X))]=\sum _{k=0}^n\phi (k)(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}=(1-p{)}^n\sum _{k=0}^n\phi (k)(\genfrac{}{}{0ex}{}{n}{k})r^k,$$with $r=p/(1-p)$. Since the right hand side is a polynomial in $r>0$ that is equal to zero, each coefficient must be zero as well, implying $\phi (k)=0$ for all $k$.

For the estimation of the parameter $p$, it is now easy to see that the UMVUE is the sample mean$$\bar{X}=\frac{1}{n}\sum _{i=1}^n{X}_i=\frac{1}{n}T(X),$$since it is unbiased and a function of $T(X)$.

Finding the UMVUE for the parameter $p(1-p)$ (that is, the variance of the distribution) is less obvious. According to method 1, we seek a function $h$ such that $$p(1-p)={\mathbb{𝔼}}_p[h(T(X))]=\sum _{k=0}^{n}h(k)(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}.$$By defining $\rho =p/(1-p)$, the above rewrites as$$\sum _{k=0}^{n}h(k)(\genfrac{}{}{0ex}{}{n}{k}){\rho }^k=\rho (1+\rho {)}^{n-2}=\sum _{k=1}^{n-1}(\genfrac{}{}{0ex}{}{n-1}{k-1}){\rho }^k.$$Comparing the coefficients shows that $h(k)=k(n-k)/(n(n-1))$, thus, the UMVUE is given by$$h(T(X))=\frac{n}{n-1}\bar{X}(1-\bar{X}).$$Noting that in the Bernoulli model $X_i=X_i^2$ and that $\sum _{i=1}^n(X_i-\bar{X}{)}^2=\sum _{i=1}^n{X}_i^2-n\stackrel{2}{\stackrel{‾}{X}}$, the UMVUE is, in fact, the unbiased sample variance $S_n^2$.

Uniform distribution

Let $X_1,\dots ,X_n$ be uniformly distributed on the interval $(0,\theta )$ for some $\theta$ to be determined. The joint probability mass function $f_p$ is of the form$$f_p(x_1,\dots ,x_n)=\prod _{i=1}^n\frac{1}{\theta }{1}_{(0,\theta )}(x_i)=\frac{1}{{\theta }^n}{1}_{[0,\mathrm{\infty })}\left(\underset{i=1,\dots ,n}{\min }{x}_i\right)1_{(-\mathrm{\infty },\theta ]}\left(\underset{i=1,\dots ,n}{\max }{x}_i\right),x\in {\mathbb{ℝ}}^n,$$thus, by the Fisher–Neyman factorization theorem, $T(x)=\underset{i=1,\dots ,n}{\max }{x}_i$ is a sufficient statistic.

To see the completeness of $T$, the distribution of $T(X)$ is

$${\mathbb{\mathbb{P}}}_{\theta }(T(X)\le t)={\mathbb{\mathbb{P}}}_{\theta }(X_i\le t,\mathrm{\forall }i=1,\dots ,n)=\prod _{i=1}^n{\mathbb{\mathbb{P}}}_{\theta }(X_i\le t)=\prod _{i=1}^n\frac{t}{\theta }=\frac{t^n}{{\theta }^n}.$$

Thus, for any measurable function $\phi$ such that ${\mathbb{𝔼}}_{\theta }[\phi (T(X))]=0$ it holds for any $\theta >0$

$$0={\mathbb{𝔼}}_{\theta }[\phi (T(X))]=\underset{0}{\overset{\theta }{\int }}\phi (x)\frac{n}{{\theta }^n}{x}^{n-1}\mathrm{d}x⟹\underset{0}{\overset{\theta }{\int }}\phi (x)x^{n-1}\mathrm{d}x=0,$$

and consequently for any $b>a>0$

$$\underset{a}{\overset{b}{\int }}\phi (x)x^{n-1}\mathrm{d}x=\underset{0}{\overset{b}{\int }}\phi (x)x^{n-1}\mathrm{d}x-\underset{0}{\overset{a}{\int }}\phi (x)x^{n-1}\mathrm{d}x=0.$$

This implies that $\phi =0$ (almost everywhere) on $[0,\mathrm{\infty })$. So $T$ is also complete. Its expectation is

$${\mathbb{𝔼}}_{\theta }[T(X)]=\underset{0}{\overset{\theta }{\int }}x\frac{n}{{\theta }^n}{x}^{n-1}\mathrm{d}x=\frac{n}{{\theta }^n}\frac{{\theta }^{n+1}}{n+1}=\frac{n}{n+1}\theta .$$

Thus, the rescaled estimator

$$T^{\ast }(X)=\frac{n+1}{n}T(X)=\frac{n+1}{n}\underset{i=1,\dots ,n}{\max }{X}_i$$

is unbiased and since it is a function of $T$, it is the UMVUE for $\theta$.

Exponential distribution

Let $X_1,\dots ,X_n$ be exponentially distributed with parameter $\lambda$ and suppose the parameter $g(\lambda )={\mathbb{\mathbb{P}}}_{\lambda }(X_1>t)$ should be estimated for some fixed $t>0$. A straightforward unbiased estimator for $g(\lambda )$ would be $\widehat{g(\lambda )}=\frac{1}{n}\sum _{i=1}^n{1}_{\{X_i>t\}}$, however, it turns out to be sub-optimal.

Since the exponential distribution is an exponential family, it is known that its sufficient statistic $\bar{X}=\frac{1}{n}\sum _{i=1}^n{X}_i$ is also complete. Moreover, the estimator $T=1_{\{X_1>t\}}$ is unbiased. Thus, according to method 2, the estimator $T^{\ast }=\mathbb{𝔼}[T\mid \bar{X}]=\mathbb{\mathbb{P}}(X_1>t\mid \bar{X})$ is the UMVUE. It remains to evaluate the conditional expectation.

First of all, $X_1/(n\bar{X})\sim \mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}(1,n-1)$, which is independent of $\lambda$ (also called ancillary). In fact, since $X_1$ and $S=\sum _{i=2}^n{X}_i$ are independent and $S\sim \mathrm{\Gamma }(n-1,\lambda )$, the cumulative distribution function at $y\in (0,1)$ reads as

$$\begin{array}{rl}\mathbb{\mathbb{P}}(\frac{X_1}{n\bar{X}}\le y)& =\mathbb{\mathbb{P}}(\frac{X_1}{X_1+S}\le y)\\ & =\underset{0}{\overset{\mathrm{\infty }}{\int }}\underset{0}{\overset{\mathrm{\infty }}{\int }}{1}_{\{x/(x+s)\le y\}}\lambda e^{-\lambda x}\frac{{\lambda }^{n-1}}{(n-2)!}{s}^{n-2}{e}^{-\lambda s}\mathrm{d}x\mathrm{d}s\\ & =\underset{0}{\overset{1}{\int }}\underset{0}{\overset{\mathrm{\infty }}{\int }}{1}_{\{u\le y\}}\frac{{\lambda }^n}{(n-2)!}{r}^{n-2}(1-u{)}^{n-2}{e}^{-\lambda r}r\mathrm{d}r\mathrm{d}u\\ & =\underset{0}{\overset{1}{\int }}{1}_{\{u\le y\}}(1-u{)}^{n-2}\frac{1}{(n-2)!}\underset{=(n-1)!}{\underbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}{\lambda }^n{r}^{n-1}{e}^{-\lambda r}\mathrm{d}r}}\mathrm{d}u\\ & =\underset{0}{\overset{y}{\int }}(n-1)(1-u{)}^{n-2}\mathrm{d}u,\end{array}$$

with the substitution $x=ru$ and $s=r(1-u)$. Thus, by Basu's theorem, $X_1/(n\bar{X})$ and $\bar{X}$ are independent and the conditional expectation simplifies to

$$\mathbb{\mathbb{P}}(X_1>t\mid \bar{X}=x)=\mathbb{\mathbb{P}}(\frac{X_1}{n\bar{X}}>\frac{t}{n\bar{X}}\mid \bar{X}=x)=\mathbb{\mathbb{P}}(\frac{X_1}{n\bar{X}}>\frac{t}{nx})={(1-\frac{t}{nx})}^{n-1}.$$

due to the above derivation. The constructed estimator

$$T^{\ast }={(1-\frac{t}{n\bar{X}})}^{n-1}$$

is unbiased and has a smaller variance than the naive estimator $\widehat{g(\lambda )}$ for all possible $\lambda >0$.

Counterexample with incomplete statistics

An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016. Let $X_1,\dots ,X_n$ be a random sample from a scale-uniform distribution $X\sim U((1-k)\theta ,(1+k)\theta ),$ with unknown mean $\mathrm{E}[X]=\theta$ and known design parameter $k\in (0,1)$. In the search for "best" possible unbiased estimators for $\theta$, it is natural to consider $X_1$ as an initial (crude) unbiased estimator for $\theta$ and then try to improve it. Since $X_1$ is not a function of $T=(X_{(1)},X_{(n)})$, the minimal sufficient statistic for $\theta$ (where $X_{(1)}=\underset{i}{\min }{X}_i$ and $X_{(n)}=\underset{i}{\max }{X}_i$), it may be improved using the Rao–Blackwell theorem as follows:

${\hat{\theta }}_{RB}={\mathrm{E}}_{\theta }[X_1\mid X_{(1)},X_{(n)}]=\frac{X_{(1)}+X_{(n)}}{2}.$

However, the following unbiased estimator can be shown to have lower variance:

${\hat{\theta }}_{LV}=\frac{1}{k^2\frac{n-1}{n+1}+1}\cdot \frac{(1-k)X_{(1)}+(1+k)X_{(n)}}{2}.$

And in fact, it could be even further improved when using the following estimator:

${\hat{\theta }}_{\text{BAYES}}=\frac{n+1}{n}[1-\frac{\frac{X_{(1)}(1+k)}{X_{(n)}(1-k)}-1}{{\left(\frac{X_{(1)}(1+k)}{X_{(n)}(1-k)}\right)}^{n+1}-1}]\frac{X_{(n)}}{1+k}$

The model is a scale model. Optimal equivariant estimators can then be derived for loss functions that are invariant.

See also

• Completeness (statistics)
• Sufficient statistic
• Minimum-variance unbiased estimator
• Rao–Blackwell theorem

References

• Casella, George; Berger, Roger L. (2002). Statistical Inference (2nd ed.). Duxbury. ISBN 0-534-24312-6. 
• Czado, Claudia; Schmidt, Thorsten (2011) (in de). Mathematische Statistik. Springer. ISBN 978-3-642-17261-8. 
• Lehmann, Erich Leo; Casella, George (1998). Theory of Point Estimation (2nd ed.). New York: Springer. ISBN 0-387-98502-6. 
• Shao, Jun (2003). Mathematical Statistics (2nd ed.). Springer. ISBN 978-0-387-95382-3.