Scoring algorithm

Short description: Form of Newton's method used in statistics

Scoring algorithm, also known as Fisher's scoring, is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.

Sketch of derivation

Let $Y_1,\dots ,Y_n$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) ${\theta }^{*}$ of $\theta$. First, suppose we have a starting point for our algorithm ${\theta }_0$, and consider a Taylor expansion of the score function, $V(\theta )$, about ${\theta }_0$:

$V(\theta )\approx V({\theta }_0)-{𝒥}({\theta }_0)(\theta -{\theta }_0),$

where

${𝒥}({\theta }_0)=-\sum _{i=1}^n{\mathrm{\nabla }{\mathrm{\nabla }}^{\mathrm{\top }}|}_{\theta ={\theta }_0}\log f(Y_i;\theta )$

is the observed information matrix at ${\theta }_0$. Now, setting $\theta ={\theta }^{*}$, using that $V({\theta }^{*})=0$ and rearranging gives us:

${\theta }^{*}\approx {\theta }_0+{{𝒥}}^{-1}({\theta }_0)V({\theta }_0).$

We therefore use the algorithm

${\theta }_{m+1}={\theta }_m+{{𝒥}}^{-1}({\theta }_m)V({\theta }_m),$

and under certain regularity conditions, it can be shown that ${\theta }_m\to {\theta }^{*}$.

Fisher scoring

In practice, ${𝒥}(\theta )$ is usually replaced by ${ℐ}(\theta )=\mathrm{E}[{𝒥}(\theta )]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

${\theta }_{m+1}={\theta }_m+{{ℐ}}^{-1}({\theta }_m)V({\theta }_m)$..

Under some regularity conditions, if ${\theta }_m$ is a consistent estimator, then ${\theta }_{m+1}$ (the correction after a single step) is 'optimal' in the sense that its error distribution is asymptotically identical to that of the true max-likelihood estimate.

See also

• Score (statistics)
• Score test
• Fisher information

Further reading

• Jennrich, R. I.; Sampson, P. F. (1976). "Newton-Raphson and Related Algorithms for Maximum Likelihood Variance Component Estimation". Technometrics 18 (1): 11–17. doi:10.1080/00401706.1976.10489395. https://www.tandfonline.com/doi/abs/10.1080/00401706.1976.10489395.