Descent direction

In optimization, a descent direction is a vector $\mathbf{𝐩}\in {\mathbb{ℝ}}^n$ that points towards a local minimum ${\mathbf{𝐱}}^{*}$ of an objective function $f:{\mathbb{ℝ}}^n\to \mathbb{ℝ}$.

Computing ${\mathbf{𝐱}}^{*}$ by an iterative method, such as line search defines a descent direction ${\mathbf{𝐩}}_k\in {\mathbb{ℝ}}^n$ at the $k$th iterate to be any ${\mathbf{𝐩}}_k$ such that $⟨{\mathbf{𝐩}}_k,\mathrm{\nabla }f({\mathbf{𝐱}}_k)⟩<0$, where $⟨,⟩$ denotes the inner product. The motivation for such an approach is that small steps along ${\mathbf{𝐩}}_k$ guarantee that $f$ is reduced, by Taylor's theorem.

Using this definition, the negative of a non-zero gradient is always a descent direction, as $⟨-\mathrm{\nabla }f({\mathbf{𝐱}}_k),\mathrm{\nabla }f({\mathbf{𝐱}}_k)⟩=-⟨\mathrm{\nabla }f({\mathbf{𝐱}}_k),\mathrm{\nabla }f({\mathbf{𝐱}}_k)⟩<0$.

Numerous methods exist to compute descent directions, all with differing merits, such as gradient descent or the conjugate gradient method.

More generally, if $P$ is a positive definite matrix, then $p_k=-P\mathrm{\nabla }f(x_k)$ is a descent direction at $x_k$. This generality is used in preconditioned gradient descent methods.

See also

• Directional derivative