Week 4: The Spectral Theorem#
Key Terms#
Symmetric and Hermitian matrices
Orthogonal diagonalization
Preparation and Syllabus#
Long Day: The rest of Chapter 2
Short Day: Theme Exercise 2
Python demo
Exercises – Long Day#
1: Types of Matrices#
Consider the matrices:
Determine for each matrix whether it is symmetric, Hermitian, and/or normal. You may use SymPy to determine normality of the matrices. For your convenience the matrices are written in code here:
A = Matrix.diag(1, 2, 3)
B = Matrix([[1, 2, 3], [3, 1, 2], [2, 3, 1]])
C = Matrix([[1, 2 + I, 3*I], [2 - I, 1, 2], [-3*I, 2, 1]])
D = Matrix([[I, 2, 3], [2, I, 2], [3, 2, I]])
Answer
\(A\) is real and a diagonal matrix and is thus automatically symmetric, Hermitian, and normal.
Answer
\(B\) is normal but not symmetric nor Hermitian.
Answer
\(C\) is Hermitian and thus also normal, but not symmetric.
Answer
\(D\) is symmetric and normal, but not Hermitian (it must be real in the diagonal).
2: Hermitian 2-by-2 Matrix. By Hand#
We consider the Hermitian matrix \(A\) given by:
This exercise concerns the computation of a spectral decomposition of \(A\), which we know exists according to the Spectral Theorem (the complex case). We will find this decomposition of \(A\) in three steps.
Question a#
Find all eigenvalues and corresponding eigenvectors of \(A\). Check your answer with SymPy’s A.eigenvects().
Answer
\(\lambda_1 = -1\) with eigenvector \(\pmb{v}_1 = [-i,1]^T\).
\(\lambda_2 = 1\) with eigenvector \(\pmb{v}_2 = [i,1]^T\).
Remember that eigenvectors may be multiplied by an arbitrary complex number that is not zero. Hence for instance \(\pmb{v}_1 = [1,i]^T\) (we have multiplied by \(i\)) is also an eigenvector corresponding to \(\lambda_1 = -1\).
Question b#
Determine an orthonormal basis consisting of eigenvectors of \(A\).
Hint
Since \(A\) is Hermitian we know that eigenvectors corresponding to different eigenvalues are orthogonal.
Hint
Hence the two eigenvalues just have to be normalized.
Question c#
This result applies to general \(n \times n\) matrices. Show that \(A = U \Lambda U^*\) if and only if \(\Lambda = U^* A U\), when \(U\) is unitary.
Hint
Multiply the equation \(A = U \Lambda U^*\) through by \(U^*\) from the left and \(U\) from the right.
Hint
Use the fact that \(U^* U = I\).
Question d#
Write out a unitary matrix \(U\) and a diagonal matrix \(\Lambda\) such that \(A = U \Lambda U^*\). This formula is called a spectral composition of \(A\). Check your result using the SymPy command:
A = Matrix([[0, I], [-I, 0]])
A.diagonalize(normalize = True)
Hint
First find \(U\). The columns of \(U\) must constitute an orthonormal basis consisting of eigenvectors.
Hint
\(\Lambda\) must contain eigenvalues in the diagonal. Their order must “match” the order of corresponding eigenvectors. If in doubt, then compute \(\Lambda\) via \(\Lambda= U^* A U\).
3: Symmetric 3-by-3 Matrix#
We are given the real and symmetric matrix
Find a spectral decomposition of \(A = Q \Lambda Q^T\). In other words, state a real orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) such that
or, equivalently,
applies. Like in the previous question we know that it exists due to the Spectral Theorem (the real case).
Hint
Find a matrix \(V\), whose columns are eigenvectors of \(A\).
Hint
An obvious possibility is
which fulfills that \( V^{-1}\, A\, V=\Lambda\), with corresponding diagonal matrix
But \(V\) is not orthogonal?
Hint
\(V\) can be orthogonalized by use of the Gram-Schmidt algorithm on its columns.
Hint
Since \(A\) is symmetric, its eigenspaces are orthogonal. Hence, it is only the two-dimensional eigenvectorspace that you must perform the Gram-Schmidt procedure on in this exercise. The one-dimensional eigenvectorspace just have to be normalized.
Answer
This is one among many possibilities. For the above choice the diagonal matrix becomes \(\Lambda = \mathrm{diag}(-4,-1,-1)\). If you are in doubt about the order of the eigenvalues in \(\Lambda\), such compute \(\Lambda = Q^T A Q\).
4: Spectral Decomposition with SymPy#
We consider the following matrices given in SymPy:
A = Matrix([[1, -1, 0, 0], [0, 1, -1, 0], [0, 0, 1, -1], [-1, 0, 0, 1]])
B = Matrix([[1, 2, 3, 4], [4, 1, 2, 3], [3, 4, 1, 2], [2, 3, 4, 1]])
A, B
We are informed that both matrices are real, normal matrices. This can be checked using:
A.conjugate() == A, B.conjugate() == B, A*A.T == A.T*A, B*B.T == B.T*B
(True, True, True, True)
We are furthermore informed that their eigenvalues are, respectively:
A.eigenvals(multiple=True), B.eigenvals(multiple=True)
Question a#
Will the below SymPy commands give us the matrices that are involved in the spectral decompositions of \(A\) and \(B\)? The call A.diagonalize(normalize = True) returns \((V,\Lambda)\) where \(A = V \Lambda V^{-1}\) with normalized eigenvectors in \(V\) and eigenvalues of \(A\) in the diagonal matrix \(\Lambda\) (according to the eigenvalue problem from Mathematics 1a).
A.diagonalize(normalize = True), B.diagonalize(normalize = True)
Question b#
Does a unitary matrix exist which diagonalizes both \(A\) and \(B\)? Meaning, does one unitary matrix exists such that \(A = U \Lambda_1 U^*\) and \(B = U \Lambda_2 U^*\), where \(\Lambda_1\) is the diagonal matrix with eigenvalues of \(A\) and \(\Lambda_2\) the diagonal matrix with eigenvalues of \(B\)?
Hint
How are the two unitary matrices from the previous question related?
Answer
Both unitary matrices from the previous question work.
Question c#
You have seen the matrix \(U^*\) before (possibly with another column order). What kind of matrix is this?
Answer
See exercise 4 from Short Day in week 3 and see this example. The matrices \(A\) and \(B\) are so-called circulant matrices which are mapping matrices for (periodic) convolution used in for example convolution neural networks (CNN). All such matrices have the Fourier matrix as eigenvector matrix \(U\).
5: Diagonalization and Reduction of Quadratic Form#
We consider the function \(q : \mathbb{R}^3 \to \mathbb{R}\) given by
Note that \(q\) can be split into two parts: a “clean” part with all second-degree terms, \(k(x,y,z)=-2x^2-2y^2-2z^2+2xy+2xz-2yz\), and a part with all terms of less than second degree, meaning the first-degree polynomial \(2x+y+z+5\).
We are given the symmetric matrix
Question a#
State a real, orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) such that
You must choose \(Q\) so it has \(\mathrm{det}\,Q=1\). You may use SymPy for this exercise.
Note
Real, orthogonal matrices always have \(\mathrm{det}\,Q = \pm 1\) (why though?), so if your choice \(Q\) has \(\mathrm{det}\,Q = - 1\) then you can just change the sign of an arbitrary column or row. Real, orthogonal matrices with \(\mathrm{det}\,Q = 1\) are said to be given a usual orientation. In \(\mathbb{R}^3\) this just means that the orthonormal basis in \(Q\) establishes a right-handed coordinate system. This does not play any big role for us in this exercise.
Hint
With SymPy you see that \(A\) has \(-4\) as a single-eigenvalue, but \(-1\) as a double-eigenvalue. Thus you must do a bit more work to find an orthonormal basis for \(\mathbb{R}^3\) consisting of eigenvectors of \(A\).
Hint
You can use SymPy’s GramSchmidt on the eigenvectors from A.eigenvects(). But maybe you can do “guessing” instead: Choose an eigenvector from the eigenspace \(E_{-4}\) and one from \(E_{-1}\). Guess on a vector that is perpendicular to them both. Normalize them all and you have three useable vectors for setting up \(Q\) with.
Answer
There are several possibilities. For instance, we can choose:
\(Q=\begin{bmatrix} -\frac{\sqrt 3}{3} & \frac{\sqrt 2}{2} & \frac{\sqrt 6}{6} \\ \frac{\sqrt 3}{3} & 0 & \frac{\sqrt 6}{3} \\ \frac{\sqrt 3}{3} & \frac{\sqrt 2}{2} & -\frac{\sqrt 6}{6} \end{bmatrix}\) og \(\Lambda=\begin{bmatrix} -4 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & -1 \end{bmatrix}.\)
Question b#
Determine the expression \(k(x,y,z),\) rewrite it to a matrix form, and reduce it.
Hint
Reducing means: Find an orthonormal basis (with usual orientation) for \(\mathbb{R}^3\) in which the expression for \(k\) does not contain mixed terms.
Hint
You may have noticed that the matrix that is a part of the matrix form of \(k(x,y,z)\) is identical to \(A\). So, you can just use the basis with corresponding diagonalization that you created above.
Answer
If we choose columns in \(Q\) as a new orthonormal basis \(\beta\), we achieve this reduction of \(k:\)
Question c#
Find an orthonormal basis with usual orientation for \(\mathbb{R}^3\) in which the expression for \(q\) does not have mixed terms. Determine its expression.
Hint
If we use the orthonormal basis \(\beta\) found above, the quadratic form is handled. Now the question is just how the first-degree polynomial that is also a part of \(q\) behaves in this new basis.
Hint
Use \(Q = {}_e[\mathrm{id}]_\beta\) found above as your change-of-basis matrix, where \(e\) denotes the standard basis. Note that \(Q^T = {}_\beta[\mathrm{id}]_e\).
Answer
If we choose the columns in \(Q\) to be the new orthonormal basis \(q\), then \(q\) gets this form:
6: Standard Equation for the three Typical Conic Sections#
In the following examples we will look at quadratic forms without mixed terms (as learned in the previous exercise we are able to get rid of these via diagonalization). Here it is possible to go one step further and remove the first-degree terms. This technique is called completing the square. In the following we will be using the technique on our journey towards identification of so-called conic sections.
Question a#
An ellipsis in the \((x,y)\) plane centred at \((c_1,c_2)\) with semi-axes \(a\) and \(b\) and with symmetry axes \(x=c_1\) and \(y=c_2\) has the standard equation
An ellipsis is given by the equation
Use the technique for completing the square, bring the equations to the standard form, and state the centre, semi-axes, and symmetry axes of the ellipsis.
Hint
Plot the given equation using the SymPy command dtuplot.plot_implicit and check your results.
Answer
The centre is \((-1,3).\) Semi-axes are \(a=1,b=2\). Symmetry axes are \(x=-1,y=3.\)
Question b#
A hyperbola in the \((x,y)\) plane centred at \((c_1,c_2)\) with semi-axes \(a\) and \(b\) and with symmetry axes \(x=c_1\) and \(y=c_2\) has the standard equation
Or alternatively (If it is not horizontal but vertical):
A hyperbola is given by the equation
Complete the square, bring the equation to standard form, and state the centre, semi-axes, and symmetry axes of the hyperbola.
Hint
Plot the given equation with the SymPy command dtuplot.plot_implicit and check your results.
Answer
Its centre is \((2,-2).\) Semi-axes are \(a=2,b=2\). Symmetry axes are \(x=2,y=-2.\)
Question c#
A parabola in the \((x,y)\) plane with vertex (stationary point) \((c_1,c_2)\) and symmetry axis \(x=c_1\) has the standard equation
Or alternatively, if the parabola is not vertical but horionzontal, which will give the symmetry axis \(y=c_2\):
A parabola is given by the equation
Complete the square, bring the equation to standard form, and state the vertex and symmetry axis of the parabola.
Hint
Plot the given equation with the SymPy dtuplot.plot_implicit, and check your results.
Answer
Vertex is at \((-3,-1).\) Symmetry axis is \(x=-3.\)
7: The Partial Derivative Increases/Decreases the most in the Gradient Direction#
This exercise is from the Notes, and its purpose is to argue for why one in the gradient method moves in the direction of the gradient.
Let \(f: \mathbb{R}^{n} \to \mathbb{R}\) be a function, for which all directional derivatives exist in \(\pmb{x} \in \mathbb{R}^{n}\). Assume that \(\nabla f(\pmb{x})\) is not the zero vector.
Question a#
Show that \(\pmb{u} := \nabla f(\pmb{x}) / \Vert \nabla f(\pmb{x}) \Vert\) is a unit vector.
Question b#
Show that the scalar \(|\nabla_{\pmb{v}}f(\pmb{x})|\) becomes largest possible, when \(\pmb{v} = \pm \pmb{u}\).
Hint
Remember that \(\nabla_{\pmb{v}}f(\pmb{x})) = \langle \pmb{v}, \nabla f (\pmb{x}) \rangle\). What do you get if you insert \(\pmb{u} = \nabla f(\pmb{x}) / \Vert \nabla f(\pmb{x}) \Vert\)?
Hint
Use Cauchy/Schwarz’ inequality theorem to argue that \(|\nabla_{\pmb{v}}f(\pmb{x})|\) cannot become any larger.
8: General Symmetric 2-by-2 Matrix#
We consider an arbitrary \(2 \times 2\) matrix. Such a matrix can be written as:
where \(a,b,\) and \(c\) are real numbers. (Why is this the case?)
Question a#
Show that the eigenvalues of \(A\) are real.
Hint
Is \(A\) Hermitian? There is a result in the Notes in Section 2.8 that you can use directly.
Hint
One can alternatively look at the sign of the discriminant of the characteristic polynomial, and then conclude that there are only real roots.
Answer
It follows directly from Lemma 2.8.1, since \(A\) is Hermitian.
The argument via the characteristic polynomial is as follows: We have that
Since the discriminant of this equation is
then \(A\) will have two real roos (counted with multiplicities). Note that the reverse does not apply: Two real eigenvalues acn also appear from a non-symmetric matrix.
Question b#
Show that if \(A\) is not a diagonal matrix, then it has two different (real) eigenvalues.
Hint
Again consider the disciminant of the characteristic polynomial.
Answer
The discriminant is \(0\) precisely when \(c=0\) and \(a=b\).
Thematic Exercise – Short Day#
Today we will carry out Theme Exercise 2.