View source for Conjugate gradient squared method

{{Short description|Algorithm for solving matrix-vector equations}}
In [[Numerical linear algebra|numerical linear algebra]], the '''conjugate gradient squared method (CGS)''' is an [[Iterative method|iterative]] algorithm for solving systems of linear equations of the form <math>A{\bold x} = {\bold b}</math>, particularly in cases where computing the [[Transpose|transpose]] <math>A^T</math> is impractical.<ref name="mathworld">{{cite web|title=Conjugate Gradient Squared Method|author1=Noel Black|author2=Shirley Moore|publisher=[[MathWorld|Wolfram Mathworld]]|url=https://mathworld.wolfram.com/ConjugateGradientSquaredMethod.html}}</ref> The CGS method was developed as an improvement to the [[Biconjugate gradient method|biconjugate gradient method]].<ref name="matlab">{{cite web|title=cgs|author=Mathworks|website=Matlab documentation|url=https://au.mathworks.com/help/matlab/ref/cgs.html}}</ref><ref name="vorst03">{{cite book|author=[[Biography:Henk van der Vorst|Henk van der Vorst]]|title=Iterative Krylov Methods for Large Linear Systems|chapter=Bi-Conjugate Gradients|year=2003|publisher=Cambridge University Press |isbn=0-521-81828-1}}</ref><ref name="SIAM">{{cite journal|title=CGS, A Fast Lanczos-Type Solver for Nonsymmetric Linear systems|author=Peter Sonneveld|journal=SIAM Journal on Scientific and Statistical Computing|volume=10|issue=1|pages=36–52|date=1989|url=https://www.proquest.com/docview/921988114|url-access=limited|doi=10.1137/0910004|id={{ProQuest|921988114}} }}</ref>

== Background ==
A [[System of linear equations|system of linear equations]] <math>A{\bold x} = {\bold b}</math> consists of a known [[Matrix (mathematics)|matrix]] <math>A</math> and a known vector <math>{\bold b}</math>. To solve the system is to find the value of the unknown vector <math>{\bold x}</math>.<ref name="vorst03" /><ref>{{Citation |title=Matrix Analysis and Applied Linear Algebra |pages=1-40 |access-date=2023-12-18 |archive-url=https://web.archive.org/web/20040610221137/http://www.matrixanalysis.com/Chapter1.pdf |chapter=Linear equations |chapter-url=http://www.matrixanalysis.com/Chapter1.pdf |place= Philadelphia, PA  |publisher=SIAM |doi=10.1137/1.9780898719512.ch1 |doi-broken-date=20 Jan 2024|archive-date=2004-06-10  }}</ref> A direct method for solving a system of linear equations is to take the inverse of the matrix <math>A</math>, then calculate <math>\bold x = A^{-1}\bold b</math>. However, computing the inverse is computationally expensive. Hence, iterative methods are commonly used. Iterative methods begin with a guess <math>\bold x^{(0)}</math>, and on each iteration the guess is improved. Once the difference between successive guesses is sufficiently small, the method has converged to a solution.<ref>{{cite web|title=Iterative Methods for Linear Systems|publisher=Mathworks|url=https://au.mathworks.com/help/matlab/math/iterative-methods-for-linear-systems.html}}</ref><ref>{{cite web|title=Iterative Methods for Solving Linear Systems|author=Jean Gallier|publisher=UPenn|url=https://www.cis.upenn.edu/~cis5150/cis515-12-sl5.pdf}}</ref>

As with the [[Conjugate gradient method|conjugate gradient method]], [[Biconjugate gradient method|biconjugate gradient method]], and similar iterative methods for solving systems of linear equations, the CGS method can be used to find solutions to multi-variable optimisation problems, such as [[Physics:Power-flow study|power-flow analysis]], hyperparameter optimisation, and [[Facial recognition system|facial recognition]].<ref>{{cite web|title=Conjugate gradient methods|author1=Alexandra Roberts|author2=Anye Shi|author3=Yue Sun|access-date=2023-12-26|publisher=[[Organization:Cornell University|Cornell University]]|url=https://optimization.cbe.cornell.edu/index.php?title=Conjugate_gradient_methods}}</ref>

== Algorithm ==

The algorithm is as follows:<ref>{{cite book|author1=R. Barrett|author2=M. Berry|author3=T. F. Chan|author4=J. Demmel|author5=J. Donato|author6=J. Dongarra|author7=V. Eijkhout|author8=R. Pozo|author9=C. Romine|author10=H. Van der Vorst|title=Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd Edition|publisher=SIAM|year=1994|url=https://netlib.org/linalg/html_templates/Templates.html}}</ref>

# Choose an initial guess <math>{\bold x}^{(0)}</math>
# Compute the residual <math>{\bold r}^{(0)} = {\bold b} - A{\bold x}^{(0)}</math>
# Choose <math>\tilde {\bold r}^{(0)} = {\bold r}^{(0)}</math>
# For <math>i = 1, 2, 3, \dots</math> do:
## <math>\rho^{(i-1)} = \tilde {\bold r}^{T(i-1)}{\bold r}^{(i-1)}</math>
## If <math>\rho^{(i-1)} = 0</math>, the method fails.
## If <math>i=1</math>:
### <math>{\bold p}^{(1)} = {\bold u}^{(1)} = {\bold r}^{(0)}</math>
## Else:
### <math>\beta^{(i-1)} = \rho^{(i-1)}/\rho^{(i-2)}</math>
### <math>{\bold u}^{(i)} = {\bold r}^{(i-1)} + \beta_{i-1}{\bold q}^{(i-1)}</math>
### <math>{\bold p}^{(i)} = {\bold u}^{(i)} + \beta^{(i-1)}({\bold q}^{(i-1)} + \beta^{(i-1)}{\bold p}^{(i-1)})</math>
## Solve <math>M\hat {\bold p}={\bold p}^{(i)}</math>, where <math>M</math> is a pre-conditioner.
## <math>\hat {\bold v} = A\hat {\bold p}</math>
## <math>\alpha^{(i)} = \rho^{(i-1)} / \tilde {\bold r}^T \hat {\bold v}</math>
## <math>{\bold q}^{(i)} = {\bold u}^{(i)} - \alpha^{(i)}\hat {\bold v}</math>
## Solve <math>M\hat {\bold u} = {\bold u}^{(i)} + {\bold q}^{(i)}</math>
## <math>{\bold x}^{(i)} = {\bold x}^{(i-1)} + \alpha^{(i)} \hat {\bold u}</math>
## <math>\hat {\bold q} = A\hat {\bold u}</math>
## <math>{\bold r}^{(i)} = {\bold r}^{(i-1)} - \alpha^{(i)}\hat {\bold q}</math>
## Check for convergence: if there is convergence, end the loop and return the result

== See also ==
* [[Biconjugate gradient method]]
* [[Biconjugate gradient stabilized method]]
* [[Generalized minimal residual method]]

== References ==

{{Reflist}}

[[Category:Numerical linear algebra]]
[[Category:Gradient methods]]

{{Sourceattribution|Conjugate gradient squared method}}