Numerical continuation

Numerical continuation is a method of computing approximate solutions of a system of parameterized nonlinear equations,

[math]\displaystyle{ F(\mathbf u,\lambda) = 0. }[/math]

The parameter [math]\displaystyle{ \lambda }[/math] is usually a real scalar, and the solution [math]\displaystyle{ \mathbf u }[/math] an n-vector. For a fixed parameter value [math]\displaystyle{ \lambda }[/math], [math]\displaystyle{ F(\cdot,\lambda) }[/math] maps Euclidean n-space into itself.

Often the original mapping [math]\displaystyle{ F }[/math] is from a Banach space into itself, and the Euclidean n-space is a finite-dimensional Banach space.

A steady state, or fixed point, of a parameterized family of flows or maps are of this form, and by discretizing trajectories of a flow or iterating a map, periodic orbits and heteroclinic orbits can also be posed as a solution of [math]\displaystyle{ F=0 }[/math].

Other forms

In some nonlinear systems, parameters are explicit. In others they are implicit, and the system of nonlinear equations is written

[math]\displaystyle{ F(\mathbf u) = 0 }[/math]

where [math]\displaystyle{ \mathbf u }[/math] is an n-vector, and its image [math]\displaystyle{ F(\mathbf u) }[/math] is an n-1 vector.

This formulation, without an explicit parameter space is not usually suitable for the formulations in the following sections, because they refer to parameterized autonomous nonlinear dynamical systems of the form:

[math]\displaystyle{ \mathbf u' = F(\mathbf u,\lambda). }[/math]

However, in an algebraic system there is no distinction between unknowns [math]\displaystyle{ \mathbf u }[/math] and the parameters.

Periodic motions

A periodic motion is a closed curve in phase space. That is, for some period [math]\displaystyle{ T }[/math],

[math]\displaystyle{ \mathbf u' = F(\mathbf u,\lambda),\, \mathbf u(0) = \mathbf u(T). }[/math]

The textbook example of a periodic motion is the undamped pendulum.

If the phase space is periodic in one or more coordinates, say [math]\displaystyle{ \mathbf u(t) = \mathbf u(t + \Omega) }[/math], with [math]\displaystyle{ \Omega }[/math] a vector , then there is a second kind of periodic motions defined by

[math]\displaystyle{ \mathbf u' = \mathbf F(\mathbf u,\lambda),\, \mathbf u(0) = \mathbf u(T + N.\Omega) }[/math]

for every integer [math]\displaystyle{ N }[/math].

The first step in writing an implicit system for a periodic motion is to move the period [math]\displaystyle{ T }[/math] from the boundary conditions to the ODE:

[math]\displaystyle{ \mathbf u' = T\mathbf F(\mathbf u,\lambda),\, \mathbf u(0)=\mathbf u(1 + N.\Omega). }[/math]

The second step is to add an additional equation, a phase constraint, that can be thought of as determining the period. This is necessary because any solution of the above boundary value problem can be shifted in time by an arbitrary amount (time does not appear in the defining equations—the dynamical system is called autonomous).

There are several choices for the phase constraint. If [math]\displaystyle{ \mathbf u_0(t) }[/math] is a known periodic orbit at a parameter value [math]\displaystyle{ \lambda_0 }[/math] near [math]\displaystyle{ \lambda }[/math], then, Poincaré used

[math]\displaystyle{ \langle \mathbf u(0) - \mathbf u_0(0),\mathbf F(\mathbf u_0(0),\lambda_0)\rangle = 0. }[/math]

which states that [math]\displaystyle{ \mathbf u }[/math] lies in a plane which is orthogonal to the tangent vector of the closed curve. This plane is called a Poincaré section.

For a general problem a better phase constraint is an integral constraint introduced by Eusebius Doedel, which chooses the phase so that the distance between the known and unknown orbits is minimized:

[math]\displaystyle{ \int_0^1 \langle\mathbf u(t) - \mathbf u_0(t),\mathbf F(\mathbf u_0(t),\lambda_0)\rangle dt = 0. }[/math]

Homoclinic and heteroclinic motions

Definitions

Solution component

Two solution components, one red and the other blue. Note that these two components may be connected outside the region of interest.

A solution component [math]\displaystyle{ \Gamma(\mathbf u_0,\lambda_0) }[/math] of the nonlinear system [math]\displaystyle{ F }[/math] is a set of points [math]\displaystyle{ (\mathbf u,\lambda) }[/math] which satisfy [math]\displaystyle{ F(\mathbf u,\lambda)=0 }[/math] and are connected to the initial solution [math]\displaystyle{ (\mathbf u_0,\lambda_0) }[/math] by a path of solutions [math]\displaystyle{ (\mathbf u(s),\lambda(s)) }[/math] for which [math]\displaystyle{ (\mathbf u(0),\lambda(0))=(\mathbf u_0,\lambda_0),\, (\mathbf u(1),\lambda(1)) = (\mathbf u,\lambda) }[/math] and [math]\displaystyle{ F(\mathbf u(s),\lambda(s))=0 }[/math].

Numerical continuation

A numerical continuation is an algorithm which takes as input a system of parametrized nonlinear equations and an initial solution [math]\displaystyle{ (\mathbf u_0,\lambda_0) }[/math], [math]\displaystyle{ F(\mathbf u_0,\lambda_0)=0 }[/math], and produces a set of points on the solution component [math]\displaystyle{ \Gamma(\mathbf u_0,\lambda_0) }[/math].

Regular point

A regular point of [math]\displaystyle{ F }[/math] is a point [math]\displaystyle{ (\mathbf u,\lambda) }[/math] at which the Jacobian of [math]\displaystyle{ F }[/math] is full rank [math]\displaystyle{ (n) }[/math].

Near a regular point the solution component is an isolated curve passing through the regular point (the implicit function theorem). In the figure above the point [math]\displaystyle{ (\mathbf u_0,\lambda_0) }[/math] is a regular point.

Singular point

A singular point of [math]\displaystyle{ F }[/math] is a point [math]\displaystyle{ (\mathbf u,\lambda) }[/math] at which the Jacobian of F is not full rank.

Near a singular point the solution component may not be an isolated curve passing through the regular point. The local structure is determined by higher derivatives of [math]\displaystyle{ F }[/math]. In the figure above the point where the two blue curves cross is a singular point.

In general solution components [math]\displaystyle{ \Gamma }[/math] are branched curves. The branch points are singular points. Finding the solution curves leaving a singular point is called branch switching, and uses techniques from bifurcation theory (singularity theory, catastrophe theory).

For finite-dimensional systems (as defined above) the Lyapunov-Schmidt decomposition may be used to produce two systems to which the Implicit Function Theorem applies. The Lyapunov-Schmidt decomposition uses the restriction of the system to the complement of the null space of the Jacobian and the range of the Jacobian.

If the columns of the matrix [math]\displaystyle{ \Phi }[/math] are an orthonormal basis for the null space of

[math]\displaystyle{ J=\left[ \begin{array}{cc} F_x & F_{\lambda}\\ \end{array} \right] }[/math]

and the columns of the matrix [math]\displaystyle{ \Psi }[/math] are an orthonormal basis for the left null space of [math]\displaystyle{ J }[/math], then the system [math]\displaystyle{ F(x,\lambda)=0 }[/math] can be rewritten as

[math]\displaystyle{ \left[ \begin{array}{l} (I-\Psi\Psi^T)F(x+\Phi\xi + \eta)\\ \Psi^T F(x+\Phi\xi + \eta)\\ \end{array} \right]=0, }[/math]

where [math]\displaystyle{ \eta }[/math] is in the complement of the null space of [math]\displaystyle{ J }[/math] [math]\displaystyle{ (\Phi^T\,\eta=0) }[/math].

In the first equation, which is parametrized by the null space of the Jacobian ([math]\displaystyle{ \xi }[/math]), the Jacobian with respect to [math]\displaystyle{ \eta }[/math] is non-singular. So the implicit function theorem states that there is a mapping [math]\displaystyle{ \eta(\xi) }[/math] such that [math]\displaystyle{ \eta(0)=0 }[/math] and [math]\displaystyle{ (I-\Psi\Psi^T)F(x+\Phi\xi+\eta(\xi))=0) }[/math]. The second equation (with [math]\displaystyle{ \eta(\xi) }[/math] substituted) is called the bifurcation equation (though it may be a system of equations).

The bifurcation equation has a Taylor expansion which lacks the constant and linear terms. By scaling the equations and the null space of the Jacobian of the original system a system can be found with non-singular Jacobian. The constant term in the Taylor series of the scaled bifurcation equation is called the algebraic bifurcation equation, and the implicit function theorem applied the bifurcation equations states that for each isolated solution of the algebraic bifurcation equation there is a branch of solutions of the original problem which passes through the singular point.

Another type of singular point is a turning point bifurcation, or saddle-node bifurcation, where the direction of the parameter [math]\displaystyle{ \lambda }[/math] reverses as the curve is followed. The red curve in the figure above illustrates a turning point.

Particular algorithms

Natural parameter continuation

Most methods of solution of nonlinear systems of equations are iterative methods. For a particular parameter value [math]\displaystyle{ \lambda_0 }[/math] a mapping is repeatedly applied to an initial guess [math]\displaystyle{ \mathbf u_0 }[/math]. If the method converges, and is consistent, then in the limit the iteration approaches a solution of [math]\displaystyle{ F(\mathbf u,\lambda_0)=0 }[/math].

Natural parameter continuation is a very simple adaptation of the iterative solver to a parametrized problem. The solution at one value of [math]\displaystyle{ \lambda }[/math] is used as the initial guess for the solution at [math]\displaystyle{ \lambda+\Delta \lambda }[/math]. With [math]\displaystyle{ \Delta \lambda }[/math] sufficiently small the iteration applied to the initial guess should converge.

One advantage of natural parameter continuation is that it uses the solution method for the problem as a black box. All that is required is that an initial solution be given (some solvers used to always start at a fixed initial guess). There has been a lot of work in the area of large scale continuation on applying more sophisticated algorithms to black box solvers (see e.g. LOCA).

However, natural parameter continuation fails at turning points, where the branch of solutions turns round. So for problems with turning points, a more sophisticated method such as pseudo-arclength continuation must be used (see below).

Simplicial or piecewise linear continuation

Simplicial Continuation, or Piecewise Linear Continuation (Allgower and Georg) is based on three basic results.

The first is

If F(x) maps IR^n into IR^(n-1), there is a unique linear interpolant on an (n-1)-dimensional simplex which agrees with the function values at the vertices of the simplex.

The second result is:

An (n-1)-dimensional simplex can be tested to determine if the unique linear interpolant takes on the value 0 inside the simplex.

Please see the article on piecewise linear continuation for details.

With these two operations this continuation algorithm is easy to state (although of course an efficient implementation requires a more sophisticated approach. See [B1]). An initial simplex is assumed to be given, from a reference simplicial decomposition of [math]\displaystyle{ \mathbb{R}^n }[/math]. The initial simplex must have at least one face which contains a zero of the unique linear interpolant on that face. The other faces of the simplex are then tested, and typically there will be one additional face with an interior zero. The initial simplex is then replaced by the simplex which lies across either face containing zero, and the process is repeated.

References: Allgower and Georg [B1] provides a crisp, clear description of the algotihm.

Pseudo-arclength continuation

This method is based on the observation that the "ideal" parameterization of a curve is arclength. Pseudo-arclength is an approximation of the arclength in the tangent space of the curve. The resulting modified natural continuation method makes a step in pseudo-arclength (rather than [math]\displaystyle{ \lambda }[/math]). The iterative solver is required to find a point at the given pseudo-arclength, which requires appending an additional constraint (the pseudo-arclength constraint) to the n by n+1 Jacobian. It produces a square Jacobian, and if the stepsize is sufficiently small the modified Jacobian is full rank.

Pseudo-arclength continuation was independently developed by Edward Riks and Gerald Wempner for finite element applications in the late 1960s, and published in journals in the early 1970s by H.B. Keller. A detailed account of these early developments is provided in the textbook by M. A. Crisfield: Nonlinear Finite Element Analysis of Solids and Structures, Vol 1: Basic Concepts, Wiley, 1991. Crisfield was one of the most active developers of this class of methods, which are by now standard procedures of commercial nonlinear finite element programs.

The algorithm is a predictor-corrector method. The prediction step finds the point (in IR^(n+1) ) which is a step [math]\displaystyle{ \Delta s }[/math] along the tangent vector at the current pointer. The corrector is usually Newton's method, or some variant, to solve the nonlinear system

[math]\displaystyle{ \begin{array}{l} F(u,\lambda)=0\\ \dot u^*_0(u-u_0)+\dot \lambda_0 (\lambda-\lambda_0) = \Delta s\\ \end{array} }[/math]

where [math]\displaystyle{ (\dot u_0,\dot\lambda_0) }[/math] is the tangent vector at [math]\displaystyle{ (u_0,\lambda_0) }[/math]. The Jacobian of this system is the bordered matrix

[math]\displaystyle{ \left[ \begin{array}{cc} F_u & F_{\lambda}\\ \dot u^* & \dot \lambda\\ \end{array} \right] }[/math]

At regular points, where the unmodified Jacobian is full rank, the tangent vector spans the null space of the top row of this new Jacobian. Appending the tangent vector as the last row can be seen as determining the coefficient of the null vector in the general solution of the Newton system (particular solution plus an arbitrary multiple of the null vector).

Gauss–Newton continuation

This method is a variant of pseudo-arclength continuation. Instead of using the tangent at the initial point in the arclength constraint, the tangent at the current solution is used. This is equivalent to using the pseudo-inverse of the Jacobian in Newton's method, and allows longer steps to be made. [B17]

Continuation in more than one parameter

The parameter [math]\displaystyle{ \lambda }[/math] in the algorithms described above is a real scalar. Most physical and design problems generally have many more than one parameter. Higher-dimensional continuation refers to the case when [math]\displaystyle{ \lambda }[/math] is a k-vector.

The same terminology applies. A regular solution is a solution at which the Jacobian is full rank [math]\displaystyle{ (n) }[/math]. A singular solution is a solution at which the Jacobian is less than full rank.

A regular solution lies on a k-dimensional surface, which can be parameterized by a point in the tangent space (the null space of the Jacobian). This is again a straightforward application of the Implicit Function Theorem.

Applications of numerical continuation techniques

Numerical continuation techniques have found a great degree of acceptance in the study of chaotic dynamical systems and various other systems which belong to the realm of catastrophe theory. The reason for such usage stems from the fact that various non-linear dynamical systems behave in a deterministic and predictable manner within a range of parameters which are included in the equations of the system. However, for a certain parameter value the system starts behaving chaotically and hence it became necessary to follow the parameter in order to be able to decipher the occurrences of when the system starts being non-predictable, and what exactly (theoretically) makes the system become unstable.

Analysis of parameter continuation can lead to more insights about stable/critical point bifurcations. Study of saddle-node, transcritical, pitch-fork, period doubling, Hopf, secondary Hopf (Neimark) bifurcations of stable solutions allows for a theoretical discussion of the circumstances and occurrences which arise at the critical points. Parameter continuation also gives a more dependable system to analyze a dynamical system as it is more stable than more interactive, time-stepped numerical solutions. Especially in cases where the dynamical system is prone to blow-up at certain parameter values (or combination of values for multiple parameters).

It is extremely insightful as to the presence of stable solutions (attracting or repelling) in the study of Nonlinear Partial Differential Equations where time stepping in the form of the Crank Nicolson algorithm is extremely time consuming as well as unstable in cases of nonlinear growth of the dependent variables in the system. The study of turbulence is another field where the Numerical Continuation techniques have been used to study the advent of turbulence in a system starting at low Reynolds numbers. Also, research using these techniques has provided the possibility of finding stable manifolds and bifurcations to invariant-tori in the case of the restricted three-body problem in Newtonian gravity and have also given interesting and deep insights into the behaviour of systems such as the Lorenz equations.

Software

(Under Construction) See also The SIAM Activity Group on Dynamical Systems' list http://www.dynamicalsystems.org/sw/sw/

• AUTO: Computation of the solutions of Two Point Boundary Value Problems (TPBVPs) with integral constraints. https://sourceforge.net/projects/auto-07p/ Available on SourceForge.
• HOMCONT: Computation of homoclinic and heteroclinic orbits. Included in AUTO
• MATCONT: Matlab toolbox for numerical continuation and bifurcation [1]Available on SourceForge.
• DDEBIFTOOL: Computation of solutions of Delay Differential Equations. A MATLAB package. Available from K. U. Leuven
• PyCont: A Python toolbox for numerical continuation and bifurcation. Native Python algorithms for fixed point continuation, sophisticated interface to AUTO for other types of problem. Included as part of PyDSTool
• CANDYS/QA: Available from the Universität Potsdam [A16]
• MANPAK: Available from Netlib [A15]
• PDDE-CONT: http://seis.bris.ac.uk/~rs1909/pdde/
• multifario: http://multifario.sourceforge.net/
• LOCA: https://trilinos.org/packages/nox-and-loca/
• DSTool
• GAIO
• OSCILL8: Oscill8 is a dynamical systems tool that allows a user to explore high-dimensional parameter space of nonlinear ODEs using bifurcation analytic techniques. Available from SourceForge.
• MANLAB : Computation of equilibrium, periodic and quasi-periodic solution of differential equations using Fourier series (harmonic balance method) developments of the solution and Taylor series developments (asymptotic numerical method) of the solution branch. Available from LMA Marseille.
• BifurcationKit.jl : This Julia package aims at performing automatic bifurcation analysis of large dimensional equations F(u,λ)=0 where λ∈ℝ by taking advantage of iterative methods, sparse formulation and specific hardwares (e.g. GPU). [2]

Examples

This problem, of finding the points which F maps into the origin appears in computer graphics as the problems of drawing contour maps (n=2), or isosurface(n=3). The contour with value h is the set of all solution components of F-h=0

References

Books

[B1] "Introduction to Numerical Continuation Methods", Eugene L. Allgower and Kurt Georg, SIAM Classics in Applied Mathematics 45. 2003.

[B2] "Numerical Methods for Bifurcations of Dynamical Equilibria", Willy J. F. Govaerts, SIAM 2000.

[B3] "Lyapunov-Schmidt Methods in Nonlinear Analysis and Applications", Nikolay Sidorov, Boris Loginov, Aleksandr Sinitsyn, and Michail Falaleev, Kluwer Academic Publishers, 2002.

[B4] "Methods of Bifurcation Theory", Shui-Nee Chow and Jack K. Hale, Springer-Verlag 1982.

[B5] "Elements of Applied Bifurcation Theory", Yuri A. Kunetsov, Springer-Verlag Applied Mathematical Sciences 112, 1995.

[B6] "Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields", John Guckenheimer and Philip Holmes, Springer-Verlag Applied Mathematical Sciences 42, 1983.

[B7] "Elementary Stability and Bifurcation Theory", Gerard Iooss and Daniel D. Joseph, Springer-Verlag Undergraduate Texts in Mathematics, 1980.

[B8] "Singularity Theory and an Introduction to Catastrophe Theory", Yung-Chen Lu, Springer-Verlag, 1976.

[B9] "Global Bifurcations and Chaos, Analytic Methods", S. Wiggins, Springer-Verlag Applied Mathematical Sciences 73, 1988.

[B10] "Singularities and Groups in Bifurcation Theory, volume I", Martin Golubitsky and David G. Schaeffer, Springer-Verlag Applied Mathematical Sciences 51, 1985.

[B11] "Singularities and Groups in Bifurcation Theory, volume II", Martin Golubitsky, Ian Stewart and David G. Schaeffer, Springer-Verlag Applied Mathematical Sciences 69, 1988.

[B12] "Solving Polynomial Systems Using Continuation for Engineering and Scientific Problems", Alexander Morgan, Prentice-Hall, Englewood Cliffs, N.J. 1987.

[B13] "Pathways to Solutions, Fixed Points and Equilibria", C. B. Garcia and W. I. Zangwill, Prentice-Hall, 1981.

[B14] "The Implicit Function Theorem: History, Theory and Applications", Steven G. Krantz and Harold R. Parks, Birkhauser, 2002.

[B15] "Nonlinear Functional Analysis", J. T. Schwartz, Gordon and Breach Science Publishers, Notes on Mathematics and its Applications, 1969.

[B16] "Topics in Nonlinear Functional Analysis", Louis Nirenberg (notes by Ralph A. Artino), AMS Courant Lecture Notes in Mathematics 6, 1974.

[B17] "Newton Methods for Nonlinear Problems -- Affine Invariance and Adaptive Algorithms", P. Deuflhard, Series Computational Mathematics 35, Springer, 2006.

Journal articles

[A1] "An Algorithm for Piecewise Linear Approximation of Implicitly Defined Two-Dimensional Surfaces", Eugene L. Allgower and Stefan Gnutzmann, SIAM Journal on Numerical Analysis, Volume 24, Number 2, 452—469, 1987.

[A2] "Simplicial and Continuation Methods for Approximations, Fixed Points and Solutions to Systems of Equations", E. L. Allgower and K. Georg, SIAM Review, Volume 22, 28—85, 1980.

[A3] "An Algorithm for Piecewise-Linear Approximation of an Implicitly Defined Manifold", Eugene L. Allgower and Phillip H. Schmidt, SIAM Journal on Numerical Analysis, Volume 22, Number 2, 322—346, April 1985.

[A4] "Contour Tracing by Piecewise Linear Approximations", David P. Dobkin, Silvio V. F. Levy, William P. Thurston and Allan R. Wilks, ACM Transactions on Graphics, 9(4) 389-423, 1990.

[A5] "Numerical Solution of Bifurcation and Nonlinear Eigenvalue Problems", H. B. Keller, in "Applications of Bifurcation Theory", P. Rabinowitz ed., Academic Press, 1977.

[A6] "A Locally Parameterized Continuation Process", W.C. Rheinboldt and J.V. Burkardt, ACM Transactions on Mathematical Software, Volume 9, 236—246, 1983.

[A7] "Nonlinear Numerics" E. Doedel, International Journal of Bifurcation and Chaos, 7(9):2127-2143, 1997.

[A8] "Nonlinear Computation", R. Seydel, International Journal of Bifurcation and Chaos, 7(9):2105-2126, 1997.

[A9] "On a Moving Frame Algorithm and the Triangulation of Equilibrium Manifolds", W.C. Rheinboldt, In T. Kuper, R. Seydel, and H. Troger eds. "ISNM79: Bifurcation: Analysis, Algorithms, Applications", pages 256-267. Birkhauser, 1987.

[A10] "On the Computation of Multi-Dimensional Solution Manifolds of Parameterized Equations", W.C. Rheinboldt, Numerishe Mathematik, 53, 1988, pages 165-181.

[A11] "On the Simplicial Approximation of Implicitly Defined Two-Dimensional Manifolds", M. L. Brodzik and W.C. Rheinboldt, Computers and Mathematics with Applications, 28(9): 9-21, 1994.

[A12] "The Computation of Simplicial Approximations of Implicitly Defined p-Manifolds", M. L. Brodzik, Computers and Mathematics with Applications, 36(6):93-113, 1998.

[A13] "New Algorithm for Two-Dimensional Numerical Continuation", R. Melville and D. S. Mackey, Computers and Mathematics with Applications, 30(1):31-46, 1995.

[A14] "Multiple Parameter Continuation: Computing Implicitly Defined k-manifolds", M. E. Henderson, IJBC 12[3]:451-76, 2003.

[A15] "MANPACK: a set of algorithms for computations on implicitly defined manifolds", W. C. Rheinboldt, Comput. Math. Applic. 27 pages 15–9, 1996.

[A16] "CANDYS/QA - A Software System For Qualitative Analysis Of Nonlinear Dynamical Systems", Feudel, U. and W. Jansen, Int. J. Bifurcation and Chaos, vol. 2 no. 4, pp. 773–794, World Scientific, 1992.