Operator norm

Short description: Measure of the "size" of linear operators

In mathematics, the operator norm measures the "size" of certain linear operators by assigning each a real number called its operator norm. Formally, it is a norm defined on the space of bounded linear operators between two given normed vector spaces. Informally, the operator norm [math]\displaystyle{ \|T\| }[/math] of a linear map [math]\displaystyle{ T : X \to Y }[/math] is the maximum factor by which it "lengthens" vectors.

Introduction and definition

Given two normed vector spaces [math]\displaystyle{ V }[/math] and [math]\displaystyle{ W }[/math] (over the same base field, either the real numbers [math]\displaystyle{ \R }[/math] or the complex numbers [math]\displaystyle{ \Complex }[/math]), a linear map [math]\displaystyle{ A : V \to W }[/math] is continuous if and only if there exists a real number [math]\displaystyle{ c }[/math] such that [math]\displaystyle{ \|Av\| \leq c \|v\| \quad \mbox{ for all } v\in V. }[/math]

The norm on the left is the one in [math]\displaystyle{ W }[/math] and the norm on the right is the one in [math]\displaystyle{ V }[/math]. Intuitively, the continuous operator [math]\displaystyle{ A }[/math] never increases the length of any vector by more than a factor of [math]\displaystyle{ c. }[/math] Thus the image of a bounded set under a continuous operator is also bounded. Because of this property, the continuous linear operators are also known as bounded operators. In order to "measure the size" of [math]\displaystyle{ A, }[/math] one can take the infimum of the numbers [math]\displaystyle{ c }[/math] such that the above inequality holds for all [math]\displaystyle{ v \in V. }[/math] This number represents the maximum scalar factor by which [math]\displaystyle{ A }[/math] "lengthens" vectors. In other words, the "size" of [math]\displaystyle{ A }[/math] is measured by how much it "lengthens" vectors in the "biggest" case. So we define the operator norm of [math]\displaystyle{ A }[/math] as [math]\displaystyle{ \|A\|_{op} = \inf\{ c \geq 0 : \|Av\| \leq c \|v\| \mbox{ for all } v \in V \}. }[/math]

The infimum is attained as the set of all such [math]\displaystyle{ c }[/math] is closed, nonempty, and bounded from below.

It is important to bear in mind that this operator norm depends on the choice of norms for the normed vector spaces [math]\displaystyle{ V }[/math] and [math]\displaystyle{ W }[/math].

Examples

Every real [math]\displaystyle{ m }[/math]-by-[math]\displaystyle{ n }[/math] matrix corresponds to a linear map from [math]\displaystyle{ \R^n }[/math] to [math]\displaystyle{ \R^m. }[/math] Each pair of the plethora of (vector) norms applicable to real vector spaces induces an operator norm for all [math]\displaystyle{ m }[/math]-by-[math]\displaystyle{ n }[/math] matrices of real numbers; these induced norms form a subset of matrix norms.

If we specifically choose the Euclidean norm on both [math]\displaystyle{ \R^n }[/math] and [math]\displaystyle{ \R^m, }[/math] then the matrix norm given to a matrix [math]\displaystyle{ A }[/math] is the square root of the largest eigenvalue of the matrix [math]\displaystyle{ A^{*} A }[/math] (where [math]\displaystyle{ A^{*} }[/math] denotes the conjugate transpose of [math]\displaystyle{ A }[/math]). This is equivalent to assigning the largest singular value of [math]\displaystyle{ A. }[/math]

Passing to a typical infinite-dimensional example, consider the sequence space [math]\displaystyle{ \ell^2, }[/math] which is an L^p space, defined by [math]\displaystyle{ l^2 = \left\{ \left(a_n\right)_{n \geq 1} : \; a_n \in \Complex, \; \sum_n |a_n|^2 \lt \infty \right\}. }[/math]

This can be viewed as an infinite-dimensional analogue of the Euclidean space [math]\displaystyle{ \Complex^n. }[/math] Now consider a bounded sequence [math]\displaystyle{ s_{\bull} = \left(s_n\right)_{n=1}^{\infty}. }[/math] The sequence [math]\displaystyle{ s_{\bull} }[/math] is an element of the space [math]\displaystyle{ \ell^{\infty}, }[/math] with a norm given by [math]\displaystyle{ \left\|s_{\bull}\right\|_{\infty} = \sup _n \left|s_n\right|. }[/math]

Define an operator [math]\displaystyle{ T_s }[/math] by pointwise multiplication: [math]\displaystyle{ \left(a_n\right)_{n=1}^{\infty} \;\stackrel{T_s}{\mapsto}\;\ \left(s_n \cdot a_n\right)_{n=1}^{\infty}. }[/math]

The operator [math]\displaystyle{ T_s }[/math] is bounded with operator norm [math]\displaystyle{ \left\|T_s\right\|_{op} = \left\|s_{\bull}\right\|_{\infty}. }[/math]

This discussion extends directly to the case where [math]\displaystyle{ \ell^2 }[/math] is replaced by a general [math]\displaystyle{ L^p }[/math] space with [math]\displaystyle{ p \gt 1 }[/math] and [math]\displaystyle{ \ell^{\infty} }[/math] replaced by [math]\displaystyle{ L^{\infty}. }[/math]

Equivalent definitions

Let [math]\displaystyle{ A : V \to W }[/math] be a linear operator between normed spaces. The first four definitions are always equivalent, and if in addition [math]\displaystyle{ V \neq \{0\} }[/math] then they are all equivalent:

[math]\displaystyle{ \begin{alignat}{4} \|A\|_{op} &= \inf &&\{ c \geq 0 ~&&:~ \| A v \| \leq c \| v \| ~&&~ \mbox{ for all } ~&&v \in V \} \\ &= \sup &&\{ \| Av \| ~&&:~ \| v \| \leq 1 ~&&~\mbox{ and } ~&&v \in V \} \\ &= \sup &&\{ \| Av \| ~&&:~ \| v \| \lt 1 ~&&~\mbox{ and } ~&&v \in V \} \\ &= \sup &&\{ \| Av \| ~&&:~ \| v \| \in \{0,1\} ~&&~\mbox{ and } ~&&v \in V \} \\ &= \sup &&\{ \| Av \| ~&&:~ \| v \| = 1 ~&&~\mbox{ and } ~&&v \in V \} \;\;\;\text{ this equality holds if and only if } V \neq \{ 0 \} \\ &= \sup &&\bigg\{ \frac{\| Av \|}{\| v \|} ~&&:~ v \ne 0 ~&&~\mbox{ and } ~&&v \in V \bigg\} \;\;\;\text{ this equality holds if and only if } V \neq \{ 0 \}. \\ \end{alignat} }[/math]

If [math]\displaystyle{ V = \{0\} }[/math] then the sets in the last two rows will be empty, and consequently their supremums over the set [math]\displaystyle{ [-\infty, \infty] }[/math] will equal [math]\displaystyle{ -\infty }[/math] instead of the correct value of [math]\displaystyle{ 0. }[/math] If the supremum is taken over the set [math]\displaystyle{ [0, \infty] }[/math] instead, then the supremum of the empty set is [math]\displaystyle{ 0 }[/math] and the formulas hold for any [math]\displaystyle{ V. }[/math]

Importantly, a linear operator [math]\displaystyle{ A : V \to W }[/math] is not, in general, guaranteed to achieve its norm [math]\displaystyle{ \|A\|_{op} = \sup \{\|A v\| : \|v\| \leq 1, v \in V\} }[/math] on the closed unit ball [math]\displaystyle{ \{v \in V : \|v\| \leq 1\}, }[/math] meaning that there might not exist any vector [math]\displaystyle{ u \in V }[/math] of norm [math]\displaystyle{ \|u\| \leq 1 }[/math] such that [math]\displaystyle{ \|A\|_{op} = \|A u\| }[/math] (if such a vector does exist and if [math]\displaystyle{ A \neq 0, }[/math] then [math]\displaystyle{ u }[/math] would necessarily have unit norm [math]\displaystyle{ \|u\| = 1 }[/math]). R.C. James proved James's theorem in 1964, which states that a Banach space [math]\displaystyle{ V }[/math] is reflexive if and only if every bounded linear functional [math]\displaystyle{ f \in V^* }[/math] achieves its norm on the closed unit ball. It follows, in particular, that every non-reflexive Banach space has some bounded linear functional (a type of bounded linear operator) that does not achieve its norm on the closed unit ball.

If [math]\displaystyle{ A : V \to W }[/math] is bounded then [math]\displaystyle{ \|A\|_{op} = \sup \left\{\left|w^*(A v)\right| : \|v\| \leq 1, \left\|w^*\right\| \leq 1 \text{ where } v \in V, w^* \in W^*\right\} }[/math] and [math]\displaystyle{ \|A\|_{op} = \left\|{}^tA\right\|_{op} }[/math] where [math]\displaystyle{ {}^t A : W^* \to V^* }[/math] is the transpose of [math]\displaystyle{ A : V \to W, }[/math] which is the linear operator defined by [math]\displaystyle{ w^* \,\mapsto\, w^* \circ A. }[/math]

Properties

The operator norm is indeed a norm on the space of all bounded operators between [math]\displaystyle{ V }[/math] and [math]\displaystyle{ W }[/math]. This means [math]\displaystyle{ \|A\|_{op} \geq 0 \mbox{ and } \|A\|_{op} = 0 \mbox{ if and only if } A = 0, }[/math] [math]\displaystyle{ \|aA\|_{op} = |a| \|A\|_{op} \mbox{ for every scalar } a , }[/math] [math]\displaystyle{ \|A + B\|_{op} \leq \|A\|_{op} + \|B\|_{op}. }[/math]

The following inequality is an immediate consequence of the definition: [math]\displaystyle{ \|Av\| \leq \|A\|_{op} \|v\| \ \mbox{ for every }\ v \in V. }[/math]

The operator norm is also compatible with the composition, or multiplication, of operators: if [math]\displaystyle{ V }[/math], [math]\displaystyle{ W }[/math] and [math]\displaystyle{ X }[/math] are three normed spaces over the same base field, and [math]\displaystyle{ A : V \to W }[/math] and [math]\displaystyle{ B : W \to X }[/math] are two bounded operators, then it is a sub-multiplicative norm, that is: [math]\displaystyle{ \|BA\|_{op} \leq \|B\|_{op} \|A\|_{op}. }[/math]

For bounded operators on [math]\displaystyle{ V }[/math], this implies that operator multiplication is jointly continuous.

It follows from the definition that if a sequence of operators converges in operator norm, it converges uniformly on bounded sets.

Table of common operator norms

By choosing different norms for the codomain, used in computing [math]\displaystyle{ \|Av\| }[/math], and the domain, used in computing [math]\displaystyle{ \|v\| }[/math], we obtain different values for the operator norm. Some common operator norms are easy to calculate, and others are NP-hard. Except for the NP-hard norms, all these norms can be calculated in [math]\displaystyle{ N^2 }[/math] operations (for an [math]\displaystyle{ N \times N }[/math] matrix), with the exception of the [math]\displaystyle{ \ell_2 - \ell_2 }[/math] norm (which requires [math]\displaystyle{ N^3 }[/math] operations for the exact answer, or fewer if you approximate it with the power method or Lanczos iterations).

Computability of Operator Norms

		Co-domain
		[math]\displaystyle{ \ell_1 }[/math]	[math]\displaystyle{ \ell_2 }[/math]	[math]\displaystyle{ \ell_\infty }[/math]
Domain	[math]\displaystyle{ \ell_1 }[/math]	Maximum [math]\displaystyle{ \ell_1 }[/math] norm of a column	Maximum [math]\displaystyle{ \ell_2 }[/math] norm of a column	Maximum [math]\displaystyle{ \ell_{\infty} }[/math] norm of a column
	[math]\displaystyle{ \ell_2 }[/math]	NP-hard	Maximum singular value	Maximum [math]\displaystyle{ \ell_2 }[/math] norm of a row
	[math]\displaystyle{ \ell_\infty }[/math]	NP-hard	NP-hard	Maximum [math]\displaystyle{ \ell_1 }[/math] norm of a row

The norm of the adjoint or transpose can be computed as follows. We have that for any [math]\displaystyle{ p, q, }[/math] then [math]\displaystyle{ \|A\|_{p\rightarrow q} = \|A^*\|_{q'\rightarrow p'} }[/math] where [math]\displaystyle{ p', q' }[/math] are Hölder conjugate to [math]\displaystyle{ p, q, }[/math] that is, [math]\displaystyle{ 1/p + 1/p' = 1 }[/math] and [math]\displaystyle{ 1/q + 1/q' = 1. }[/math]

Operators on a Hilbert space

Suppose [math]\displaystyle{ H }[/math] is a real or complex Hilbert space. If [math]\displaystyle{ A : H \to H }[/math] is a bounded linear operator, then we have [math]\displaystyle{ \|A\|_{op} = \left\|A^*\right\|_{op} }[/math] and [math]\displaystyle{ \left\|A^* A\right\|_{op} = \|A\|_{op}^2, }[/math] where [math]\displaystyle{ A^{*} }[/math] denotes the adjoint operator of [math]\displaystyle{ A }[/math] (which in Euclidean spaces with the standard inner product corresponds to the conjugate transpose of the matrix [math]\displaystyle{ A }[/math]).

In general, the spectral radius of [math]\displaystyle{ A }[/math] is bounded above by the operator norm of [math]\displaystyle{ A }[/math]: [math]\displaystyle{ \rho(A) \leq \|A\|_{op}. }[/math]

To see why equality may not always hold, consider the Jordan canonical form of a matrix in the finite-dimensional case. Because there are non-zero entries on the superdiagonal, equality may be violated. The quasinilpotent operators is one class of such examples. A nonzero quasinilpotent operator [math]\displaystyle{ A }[/math] has spectrum [math]\displaystyle{ \{0\}. }[/math] So [math]\displaystyle{ \rho(A) = 0 }[/math] while [math]\displaystyle{ \|A\|_{op} \gt 0. }[/math]

However, when a matrix [math]\displaystyle{ N }[/math] is normal, its Jordan canonical form is diagonal (up to unitary equivalence); this is the spectral theorem. In that case it is easy to see that [math]\displaystyle{ \rho(N) = \|N\|_{op}. }[/math]

This formula can sometimes be used to compute the operator norm of a given bounded operator [math]\displaystyle{ A }[/math]: define the Hermitian operator [math]\displaystyle{ B = A^{*} A, }[/math] determine its spectral radius, and take the square root to obtain the operator norm of [math]\displaystyle{ A. }[/math]

The space of bounded operators on [math]\displaystyle{ H, }[/math] with the topology induced by operator norm, is not separable. For example, consider the Lp space [math]\displaystyle{ L^2[0, 1], }[/math] which is a Hilbert space. For [math]\displaystyle{ 0 \lt t \leq 1, }[/math] let [math]\displaystyle{ \Omega_t }[/math] be the characteristic function of [math]\displaystyle{ [0, t], }[/math] and [math]\displaystyle{ P_t }[/math] be the multiplication operator given by [math]\displaystyle{ \Omega_t, }[/math] that is, [math]\displaystyle{ P_t (f) = f \cdot \Omega_t. }[/math]

Then each [math]\displaystyle{ P_t }[/math] is a bounded operator with operator norm 1 and [math]\displaystyle{ \left\|P_t - P_s\right\|_{op} = 1 \quad \mbox{ for all } \quad t \neq s. }[/math]

But [math]\displaystyle{ \{P_t : 0 \lt t \leq 1\} }[/math] is an uncountable set. This implies the space of bounded operators on [math]\displaystyle{ L^2([0, 1]) }[/math] is not separable, in operator norm. One can compare this with the fact that the sequence space [math]\displaystyle{ \ell^{\infty} }[/math] is not separable.

The associative algebra of all bounded operators on a Hilbert space, together with the operator norm and the adjoint operation, yields a C*-algebra.

See also

• Banach–Mazur compactum – Set of n-dimensional subspaces of a normed space made into a compact metric space.
• Continuous linear operator
• Contraction (operator theory) – Bounded operators with sub-unit norm
• Discontinuous linear map
• Dual norm – Measurement on a normed vector space
• Matrix norm – Norm on a vector space of matrices
• Norm (mathematics) – Length in a vector space
• Operator algebra – Branch of functional analysis
• Operator theory – Mathematical field of study
• Unbounded operator – Linear operator defined on a dense linear subspace

References

• Aliprantis, Charalambos D.; Border, Kim C. (2007), Infinite Dimensional Analysis: A Hitchhiker's Guide, Springer, p. 229, ISBN 9783540326960, https://books.google.com/books?id=4hIq6ExH7NoC&pg=PA229 .
• Conway, John B. (1990), "III.2 Linear Operators on Normed Spaces", A Course in Functional Analysis, New York: Springer-Verlag, pp. 67–69, ISBN 0-387-97245-5, https://books.google.com/books?id=ix4P1e6AkeIC&pg=PA67 
• Diestel, Joe (1984). Sequences and series in Banach spaces. New York: Springer-Verlag. ISBN 0-387-90859-5. OCLC 9556781. 
• Rudin, Walter (January 1, 1991). Functional Analysis. International Series in Pure and Applied Mathematics. 8 (Second ed.). New York, NY: McGraw-Hill Science/Engineering/Math. ISBN 978-0-07-054236-5. OCLC 21163277. https://archive.org/details/functionalanalys00rudi.