Penalty function

A technique for introducing constraints into an otherwise unconstrained minimization problem, the name comes from the idea of adding a penalty for the violation of constraints. While minimizing the function, one therefore minimizes also the constraint violation. In the limit that the penalty is large compared with the rest of the function, the constraints will eventually be satisfied if possible. The technique is very general and can be applied to both equality and inequality constraints, but is of course not as efficient as more specialized methods designed for particular types of constraints.

For equality constraints of the form g(a) = 0, where the vector a represents the free parameters of the problem, and g may be a vector if there are more constraints than one, the penalty function should be P = kg²(a), so that the total function to be minimized would be

or, more generally

where f(a) is the usual or negative log-likelihood function, and k is a positive constant chosen large enough that the penalty function is more important than f(a). For inequality constraints of the form g(a)>0, the same formalism applies, except that the penalty function is added only when the constraints are violated ( Minimization).