Normal-inverse-gamma distribution

Short description: Family of multivariate continuous probability distributions

normal-inverse-gamma

Probability density function
Parameters	$\mu$ location (real) $\lambda >0$ (real) $\alpha >0$ (real) $\beta >0$ (real)
Support	$x\in (-\mathrm{\infty },\mathrm{\infty }),{\sigma }^2\in (0,\mathrm{\infty })$
PDF	$\frac{\sqrt{\lambda }}{\sqrt{2\pi {\sigma }^2}}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1}\exp (-\frac{2\beta +\lambda (x-\mu {)}^2}{2{\sigma }^2})$
Mean	$\mathrm{E}[x]=\mu$ $\mathrm{E}[{\sigma }^2]=\frac{\beta }{\alpha -1}$, for $\alpha >1$
Mode	$x=\mu \text{<mrow data-mjx-texclass="ORD"><mo stretchy="false">(</mo>univariate<mo stretchy="false">)</mo></mrow>},x=\mathit{\mu }\text{<mrow data-mjx-texclass="ORD"><mo stretchy="false">(</mo>multivariate<mo stretchy="false">)</mo></mrow>}$ ${\sigma }^2=\frac{\beta }{\alpha +1+1/2}\text{<mrow data-mjx-texclass="ORD"><mo stretchy="false">(</mo>univariate<mo stretchy="false">)</mo></mrow>},{\sigma }^2=\frac{\beta }{\alpha +1+k/2}\text{<mrow data-mjx-texclass="ORD"><mo stretchy="false">(</mo>multivariate<mo stretchy="false">)</mo></mrow>}$
Variance	$\mathrm{Var}[x]=\frac{\beta }{(\alpha -1)\lambda }$, for $\alpha >1$ $\mathrm{Var}[{\sigma }^2]=\frac{{\beta }^2}{(\alpha -1{)}^2(\alpha -2)}$, for $\alpha >2$ $\mathrm{Cov}[x,{\sigma }^2]=0$, for $\alpha >1$

In probability theory and statistics, the normal-inverse-gamma distribution (or Gaussian-inverse-gamma distribution) is a four-parameter family of multivariate continuous probability distributions. It is the conjugate prior of a normal distribution with unknown mean and variance.

Definition

Suppose

$x\mid {\sigma }^2,\mu ,\lambda \sim \mathrm{N}(\mu ,{\sigma }^2/\lambda )$

has a normal distribution with mean $\mu$ and variance ${\sigma }^2/\lambda$, where

${\sigma }^2\mid \alpha ,\beta \sim {\mathrm{\Gamma }}^{-1}(\alpha ,\beta )$

has an inverse-gamma distribution. Then $(x,{\sigma }^2)$ has a normal-inverse-gamma distribution, denoted as

$(x,{\sigma }^2)\sim \text{N-}{\mathrm{\Gamma }}^{-1}(\mu ,\lambda ,\alpha ,\beta ).$

($\text{NIG}$ is also used instead of $\text{N-}{\mathrm{\Gamma }}^{-1}.$)

The normal-inverse-Wishart distribution is a generalization of the normal-inverse-gamma distribution that is defined over multivariate random variables.

Characterization

Probability density function

$f(x,{\sigma }^2\mid \mu ,\lambda ,\alpha ,\beta )=\frac{\sqrt{\lambda }}{\sigma \sqrt{2\pi }}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1}\exp (-\frac{2\beta +\lambda (x-\mu {)}^2}{2{\sigma }^2})$

For the multivariate form where $\mathbf{𝐱}$ is a $k\times 1$ random vector,

$f(\mathbf{𝐱},{\sigma }^2\mid \mu ,{\mathbf{𝐕}}^{-1},\alpha ,\beta )=|\mathbf{𝐕}{|}^{-1/2}(2\pi {)}^{-k/2}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1+k/2}\exp (-\frac{2\beta +(\mathbf{𝐱}-\mathit{\mu }{)}^T{\mathbf{𝐕}}^{-1}(\mathbf{𝐱}-\mathit{\mu })}{2{\sigma }^2}).$

where $|\mathbf{𝐕}|$ is the determinant of the $k\times k$ matrix $\mathbf{𝐕}$. Note how this last equation reduces to the first form if $k=1$ so that $\mathbf{𝐱},\mathbf{𝐕},\mathit{\mu }$ are scalars.

Alternative parameterization

It is also possible to let $\gamma =1/\lambda$ in which case the pdf becomes

$f(x,{\sigma }^2\mid \mu ,\gamma ,\alpha ,\beta )=\frac{1}{\sigma \sqrt{2\pi \gamma }}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1}\exp (-\frac{2\gamma \beta +(x-\mu {)}^2}{2\gamma {\sigma }^2})$

In the multivariate form, the corresponding change would be to regard the covariance matrix $\mathbf{𝐕}$ instead of its inverse ${\mathbf{𝐕}}^{-1}$ as a parameter.

Cumulative distribution function

$F(x,{\sigma }^2\mid \mu ,\lambda ,\alpha ,\beta )=\frac{e^{-\frac{\beta }{{\sigma }^2}}{\left(\frac{\beta }{{\sigma }^2}\right)}^{\alpha }(\mathrm{erf}\left(\frac{\sqrt{\lambda }(x-\mu )}{\sqrt{2}\sigma }\right)+1)}{2{\sigma }^2\mathrm{\Gamma }(\alpha )}$

Properties

Marginal distributions

Given $(x,{\sigma }^2)\sim \text{N-}{\mathrm{\Gamma }}^{-1}(\mu ,\lambda ,\alpha ,\beta ).$ as above, ${\sigma }^2$ by itself follows an inverse gamma distribution:

${\sigma }^2\sim {\mathrm{\Gamma }}^{-1}(\alpha ,\beta )$

while $\sqrt{\frac{\alpha \lambda }{\beta }}(x-\mu )$ follows a t distribution with $2\alpha$ degrees of freedom.

Proof for $\lambda =1$

For $\lambda =1$ probability density function is

$f(x,{\sigma }^2\mid \mu ,\alpha ,\beta )=\frac{1}{\sigma \sqrt{2\pi }}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1}\exp (-\frac{2\beta +(x-\mu {)}^2}{2{\sigma }^2})$

Marginal distribution over $x$ is

$\begin{array}{rl}f(x\mid \mu ,\alpha ,\beta )& =\underset{0}{\overset{\mathrm{\infty }}{\int }}d{\sigma }^{2}f(x,{\sigma }^2\mid \mu ,\alpha ,\beta )\\ & =\frac{1}{\sqrt{2\pi }}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}\underset{0}{\overset{\mathrm{\infty }}{\int }}d{\sigma }^2{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1/2+1}\exp (-\frac{2\beta +(x-\mu {)}^2}{2{\sigma }^2})\end{array}$

Except for normalization factor, expression under the integral coincides with Inverse-gamma distribution

${\mathrm{\Gamma }}^{-1}(x;a,b)=\frac{b^a}{\mathrm{\Gamma }(a)}\frac{e^{-b/x}}{x^{a+1}},$

with $x={\sigma }^2$, $a=\alpha +1/2$, $b=\frac{2\beta +(x-\mu {)}^2}{2}$.

Since $\underset{0}{\overset{\mathrm{\infty }}{\int }}dx{\mathrm{\Gamma }}^{-1}(x;a,b)=1,\underset{0}{\overset{\mathrm{\infty }}{\int }}dx{x}^{-(a+1)}{e}^{-b/x}=\mathrm{\Gamma }(a)b^{-a}$, and

$\underset{0}{\overset{\mathrm{\infty }}{\int }}d{\sigma }^2{\left(\frac{1}{{\sigma }^2}\right)}^{\alpha +1/2+1}\exp (-\frac{2\beta +(x-\mu {)}^2}{2{\sigma }^2})=\mathrm{\Gamma }(\alpha +1/2){\left(\frac{2\beta +(x-\mu {)}^2}{2}\right)}^{-(\alpha +1/2)}$

Substituting this expression and factoring dependence on $x$,

$f(x\mid \mu ,\alpha ,\beta ){\propto }_x{(1+\frac{(x-\mu {)}^2}{2\beta })}^{-(\alpha +1/2)}.$

Shape of generalized Student's t-distribution is

$t(x|\nu ,\hat{\mu },{\hat{\sigma }}^2){\propto }_x{(1+\frac{1}{\nu }\frac{(x-\hat{\mu }{)}^2}{{\hat{\sigma }}^2})}^{-(\nu +1)/2}$.

Marginal distribution $f(x\mid \mu ,\alpha ,\beta )$ follows t-distribution with $2\alpha$ degrees of freedom

$f(x\mid \mu ,\alpha ,\beta )=t(x|\nu =2\alpha ,\hat{\mu }=\mu ,{\hat{\sigma }}^2=\beta /\alpha )$.

In the multivariate case, the marginal distribution of $\mathbf{𝐱}$ is a multivariate t distribution:

$\mathbf{𝐱}\sim t_{2\alpha }(\mathit{\mu },\frac{\beta }{\alpha }\mathbf{𝐕})$

Scaling

Suppose

$(x,{\sigma }^2)\sim \text{N-}{\mathrm{\Gamma }}^{-1}(\mu ,\lambda ,\alpha ,\beta ).$

Then for $c>0$,

$(cx,c{\sigma }^2)\sim \text{N-}{\mathrm{\Gamma }}^{-1}(c\mu ,\lambda /c,\alpha ,c\beta ).$

Proof: To prove this let $(x,{\sigma }^2)\sim \text{N-}{\mathrm{\Gamma }}^{-1}(\mu ,\lambda ,\alpha ,\beta )$ and fix $c>0$. Defining $Y=(Y_1,Y_2)=(cx,c{\sigma }^2)$, observe that the PDF of the random variable $Y$ evaluated at $(y_1,y_2)$ is given by $1/c^2$ times the PDF of a $\text{N-}{\mathrm{\Gamma }}^{-1}(\mu ,\lambda ,\alpha ,\beta )$ random variable evaluated at $(y_1/c,y_2/c)$. Hence the PDF of $Y$ evaluated at $(y_1,y_2)$ is given by :$f_Y(y_1,y_2)=\frac{1}{c^2}\frac{\sqrt{\lambda }}{\sqrt{2\pi y_2/c}}\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{y_2/c}\right)}^{\alpha +1}\exp (-\frac{2\beta +\lambda (y_1/c-\mu {)}^2}{2y_2/c})=\frac{\sqrt{\lambda /c}}{\sqrt{2\pi y_2}}\frac{(c\beta {)}^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\left(\frac{1}{y_2}\right)}^{\alpha +1}\exp (-\frac{2c\beta +(\lambda /c)(y_1-c\mu {)}^2}{2y_2}).$

The right hand expression is the PDF for a $\text{N-}{\mathrm{\Gamma }}^{-1}(c\mu ,\lambda /c,\alpha ,c\beta )$ random variable evaluated at $(y_1,y_2)$, which completes the proof.

Exponential family

Normal-inverse-gamma distributions form an exponential family with natural parameters ${\theta }_1=\frac{-\lambda }{2}$, ${\theta }_2=\lambda \mu$, ${\theta }_3=\alpha$, and ${\theta }_4=-\beta +\frac{-\lambda {\mu }^2}{2}$ and sufficient statistics $T_1=\frac{x^2}{{\sigma }^2}$, $T_2=\frac{x}{{\sigma }^2}$, $T_3=\log \left(\frac{1}{{\sigma }^2}\right)$, and $T_4=\frac{1}{{\sigma }^2}$.

Kullback–Leibler divergence

Measures difference between two distributions.

Posterior distribution of the parameters

See the articles on normal-gamma distribution and conjugate prior.

Interpretation of the parameters

See the articles on normal-gamma distribution and conjugate prior.

Generating normal-inverse-gamma random variates

Generation of random variates is straightforward:

1. Sample ${\sigma }^2$ from an inverse gamma distribution with parameters $\alpha$ and $\beta$
2. Sample $x$ from a normal distribution with mean $\mu$ and variance ${\sigma }^2/\lambda$

Related distributions

• The normal-gamma distribution is the same distribution parameterized by precision rather than variance
• A generalization of this distribution which allows for a multivariate mean and a completely unknown positive-definite covariance matrix ${\sigma }^2\mathbf{𝐕}$ (whereas in the multivariate inverse-gamma distribution the covariance matrix is regarded as known up to the scale factor ${\sigma }^2$) is the normal-inverse-Wishart distribution

See also

• Compound probability distribution

References

• Denison, David G. T. et al. (2002). Bayesian Methods for Nonlinear Classification and Regression. Wiley. ISBN 0471490369. 
• Koch, Karl-Rudolf (2007). Introduction to Bayesian Statistics (2nd ed.). Springer. ISBN 354072723X.