Hoeffding's lemma
In probability theory, Hoeffding's lemma is an inequality that bounds the moment-generating function of any bounded random variable.[1] It is named after the Finnish–United States mathematical statistician Wassily Hoeffding. The proof of Hoeffding's lemma uses Taylor's theorem and Jensen's inequality. Hoeffding's lemma is itself used in the proof of McDiarmid's inequality.
Statement of the lemma
Let X be any real-valued random variable such that [math]\displaystyle{ a \leq X \leq b }[/math] almost surely, i.e. with probability one. Then, for all [math]\displaystyle{ \lambda \in \mathbb{R} }[/math],
- [math]\displaystyle{ \mathbb{E} \left[ e^{\lambda X} \right] \leq \exp \Big(\lambda\mathbb{E}[X]+ \frac{\lambda^2 (b - a)^2}{8} \Big), }[/math]
or equivalently,
- [math]\displaystyle{ \mathbb{E} \left[ e^{\lambda (X - \mathbb{E}[X])} \right] \leq \exp \Big(\frac{\lambda^2 (b - a)^2}{8} \Big). }[/math]
Proof
Without loss of generality, by replacing [math]\displaystyle{ X }[/math] by [math]\displaystyle{ X - \mathbb{E}[X] }[/math], we can assume [math]\displaystyle{ \mathbb{E}[X] = 0 }[/math], so that [math]\displaystyle{ a \leq 0 \leq b }[/math].
Since [math]\displaystyle{ e^{\lambda x} }[/math] is a convex function of [math]\displaystyle{ x }[/math], we have that for all [math]\displaystyle{ x \in [a,b] }[/math],
- [math]\displaystyle{ e^{\lambda x}\leq \frac{b-x}{b-a}e^{\lambda a}+\frac{x-a}{b-a}e^{\lambda b} }[/math]
So,
- [math]\displaystyle{ \begin{align} \mathbb{E}\left[e^{\lambda X}\right] &\leq \frac{b-\mathbb{E}[X]}{b-a}e^{\lambda a}+\frac{\mathbb{E}[X]-a}{b-a}e^{\lambda b}\\ &= \frac{b}{b-a}e^{\lambda a} + \frac{-a}{b-a}e^{\lambda b} \\ &= e^{L(\lambda(b-a))}, \end{align} }[/math]
where [math]\displaystyle{ L(h)= \frac{ha}{b-a}+\ln(1 + \frac{a - e^ha}{b-a}) }[/math]. By computing derivatives, we find
- [math]\displaystyle{ L(0)=L'(0)=0 }[/math] and [math]\displaystyle{ L''(h)= -\frac{ab e^h}{(b-a e^h)^2} }[/math].
From the AMGM inequality we thus see that [math]\displaystyle{ L''(h)\le\frac14 }[/math] for all [math]\displaystyle{ h }[/math], and thus, from Taylor's theorem, there is some [math]\displaystyle{ 0 \le \theta \le 1 }[/math] such that
- [math]\displaystyle{ L(h) = L(0) + h L'(0) + \frac{1}{2} h^2 L''(h\theta) \leq \frac{1}{8}h^2. }[/math]
Thus, [math]\displaystyle{ \mathbb{E}\left[e^{\lambda X}\right] \leq e^{\frac{1}{8}\lambda^2(b-a)^2} }[/math].
See also
Notes
- ↑ Pascal Massart (26 April 2007). Concentration Inequalities and Model Selection: Ecole d'Eté de Probabilités de Saint-Flour XXXIII - 2003. Springer. p. 21. ISBN 978-3-540-48503-2. https://books.google.com/books?id=ZI67BQAAQBAJ&pg=PA21.
 |  |
