Eigenvectors and Eigenvalues
1 Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors are special scalars and vectors (numbers and directions) that describe how a linear transformation (matrix) acts on a [[vector space]].
An eigenvector is a non-zero vector whose direction does not change when a matrix transformation is applied to it; it only gets stretched, compressed or reflected.
An eigenvalue is the factor by which an eigenvector is stretched or compressed during a matrix transformation.
2 Eigenvalue Equation
For a square matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\), a non-zero vector \(\mathbf{q}\) and scalar \(\lambda\) satisfying:
\[ \mathbf{A}\mathbf{q} = \lambda \mathbf{q} \]
are called an eigenvector and its associated eigenvalue, respectively. The eigenvalue equation says that \(\mathbf{A}\) stretches (or reflects) \(\mathbf{q}\) by factor \(\lambda\) without changing its direction.
Eigenvalues are found by solving the characteristic equation
\[ \det(\mathbf{A} - \lambda \mathbf{I}) = 0. \]
3 Eigendecomposition
A symmetric matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) always has \(n\) real eigenvalues \(\lambda_1, \ldots, \lambda_n\) and \(n\) orthonormal eigenvectors \(\mathbf{q}_1, \ldots, \mathbf{q}_n\) (by the spectral theorem). We can write:
\[ \mathbf{A} = \mathbf{Q} \boldsymbol{\Lambda} \mathbf{Q}^\top, \]
where \(\mathbf{Q} = [\mathbf{q}_1 \mid \cdots \mid \mathbf{q}_n]\) is orthogonal and \(\boldsymbol{\Lambda} = \mathrm{diag}(\lambda_1, \ldots, \lambda_n)\).
4 Trace and Determinant via Eigenvalues
Two convenient identities link eigenvalues to matrix invariants:
\[ \mathrm{tr}(\mathbf{A}) = \sum_{i=1}^n \lambda_i, \qquad \det(\mathbf{A}) = \prod_{i=1}^n \lambda_i. \]
These are useful for understanding properties of the hat matrix later.
5 Eigenvalues of a Linear Operator
The definitions above apply to any linear operator \(P:\mathcal{V}\to\mathcal{V}\) (see Linear Maps): \(\vec{x}\neq\vec{0}\) is an eigenvector with eigenvalue \(\lambda\in F\) if \(P\vec{x}=\lambda\vec{x}\). For a matrix \(A\) the eigenvalues are the roots in \(F\) of the characteristic polynomial \(c_A(\lambda)=\det(A-\lambda I_n)\), since \(A-\lambda I_n\) has a nontrivial kernel exactly when it is singular. The set \(\ker(A-\lambda I_n)\) is the eigenspace of \(\lambda\). Similar matrices have the same characteristic polynomial, so eigenvalues are properties of the operator and not of the chosen basis.
6 Diagonalisation
If \(\mathcal{V}\) has a basis \(\mathcal{B}=(\vec{x}_1,\dots,\vec{x}_n)\) of eigenvectors of \(P\), then \[ [P]_{\mathcal{B}}=\operatorname{diag}(\lambda_1,\dots,\lambda_n), \] where \(\lambda_i\) is the eigenvalue of \(\vec{x}_i\). Conversely, if \([P]_{\mathcal{B}}\) is diagonal then \(\mathcal{B}\) consists of eigenvectors.
Proof. Since \(P\vec{x}_i=\lambda_i\vec{x}_i\), the \(i\)-th column of \([P]_{\mathcal{B}}\) is \([P\vec{x}_i]_{\mathcal{B}}=\lambda_i\vec{e}_i\), with \(\lambda_i\) in position \(i\) and zeros elsewhere. The converse reads the same computation backwards.
In matrix form, if \(S\) has the eigenvectors as columns then \(A=S\Lambda S^{-1}\) with \(\Lambda=\operatorname{diag}(\lambda_i)\), by the change of basis formula. Not every matrix is diagonalisable (for instance \(\begin{bmatrix}1&1\\0&1\end{bmatrix}\) has only one eigenvector direction), but symmetric matrices always are, see Matrix Decompositions.