Matrices

1 Matrices

A matrix \(\mathbf{A} \in \mathbb{R}^{m \times n}\) is a rectangular array with \(m\) rows and \(n\) columns:

\[ \mathbf{A} = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix}. \]

Addition and scalar multiplication are element-wise (matrices must have the same shape for addition). Matrix multiplication \(\mathbf{A}\mathbf{B}\) requires the inner dimensions to match: if \(\mathbf{A} \in \mathbb{R}^{m \times k}\) and \(\mathbf{B} \in \mathbb{R}^{k \times n}\) then \(\mathbf{C} = \mathbf{A}\mathbf{B} \in \mathbb{R}^{m \times n}\) with entries:

\[ c_{ij} = \sum_{l=1}^k a_{il}\, b_{lj}. \]

Note that matrix multiplication is not commutative in general: \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\).

2 Transpose

The transpose of \(\mathbf{A} \in \mathbb{R}^{m \times n}\) is \(\mathbf{A}^\top \in \mathbb{R}^{n \times m}\), obtained by swapping rows and columns: \((A^\top)_{ij} = a_{ji}\).

Key properties of the transpose:

\[ (\mathbf{A}^\top)^\top = \mathbf{A}, \qquad (\mathbf{A} + \mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top, \qquad (\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top \mathbf{A}^\top. \]

The last identity is the reversal rule for transposes, and appears constantly in regression derivations.

3 Special Matrices

3.1 Identity Matrix

The identity matrix \(\mathbf{I}_n \in \mathbb{R}^{n \times n}\) has ones on the diagonal and zeros elsewhere. It satisfies \(\mathbf{A}\mathbf{I} = \mathbf{I}\mathbf{A} = \mathbf{A}\) for any compatible matrix \(\mathbf{A}\).

3.2 Diagonal Matrix

A diagonal matrix has non-zero entries only on the main diagonal. Diagonal matrices are easy to invert: if \(\mathbf{D} = \mathrm{diag}(d_1, \ldots, d_n)\) with all \(d_i \neq 0\), then \(\mathbf{D}^{-1} = \mathrm{diag}(1/d_1, \ldots, 1/d_n)\).

3.3 Symmetric Matrix

A matrix is symmetric if \(\mathbf{A}^\top = \mathbf{A}\), i.e. \(a_{ij} = a_{ji}\) for all \(i, j\). Many important matrices in statistics are symmetric: covariance matrices, Gram matrices \(\mathbf{X}^\top \mathbf{X}\), and the hat matrix \(\mathbf{H}\).

3.4 Orthogonal Matrix

A square matrix \(\mathbf{Q}\) is orthogonal if its columns are orthonormal (mutually orthogonal unit vectors), which implies:

\[ \mathbf{Q}^\top \mathbf{Q} = \mathbf{Q}\mathbf{Q}^\top = \mathbf{I}, \quad \text{i.e.} \quad \mathbf{Q}^{-1} = \mathbf{Q}^\top. \]

Orthogonal matrices preserve norms: \(\|\mathbf{Q}\mathbf{v}\| = \|\mathbf{v}\|\). They arise in eigendecompositions and in the geometric interpretation of least squares.

4 Rank and Invertibility

4.1 Rank

The rank of a matrix \(\mathbf{A} \in \mathbb{R}^{m \times n}\) is the maximum number of linearly independent columns (equivalently, rows). We always have \(\mathrm{rank}(\mathbf{A}) \leq \min(m, n)\).

• \(\mathbf{A}\) has full column rank if \(\mathrm{rank}(\mathbf{A}) = n\) (all columns are linearly independent).
• \(\mathbf{A}\) has full row rank if \(\mathrm{rank}(\mathbf{A}) = m\).

5 Matrix Inverse

For a square matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\), the inverse \(\mathbf{A}^{-1}\) satisfies:

\[ \mathbf{A}\mathbf{A}^{-1} = \mathbf{A}^{-1}\mathbf{A} = \mathbf{I}_n. \]

A square matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) is invertible (non-singular) if and only if any of the following equivalent conditions hold:

1. \(\mathrm{rank}(\mathbf{A}) = n\) (full rank).
2. \(\det(\mathbf{A}) \neq 0\).
3. All eigenvalues of \(\mathbf{A}\) are non-zero.
4. The columns of \(\mathbf{A}\) are linearly independent.

Key properties of the inverse:

\[ (\mathbf{A}^{-1})^{-1} = \mathbf{A}, \qquad (\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}, \qquad (\mathbf{A}^\top)^{-1} = (\mathbf{A}^{-1})^\top. \]

The reversal rule \((\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}\) mirrors the reversal rule for transposes.

6 Quadratic Forms and Positive Definiteness

A quadratic form associated with a symmetric matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) is the scalar-valued function:

\[ Q(\mathbf{v}) = \mathbf{v}^\top \mathbf{A} \mathbf{v} = \sum_{i=1}^n \sum_{j=1}^n a_{ij} v_i v_j. \]

Quadratic forms arise throughout statistics — for example, the OLS residual sum of squares is \(\mathrm{RSS} = \mathbf{e}^\top \mathbf{e}\) and the Mahalanobis distance is \((\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})\).

A symmetric matrix \(\mathbf{A}\) is:

• Positive definite (PD) if \(\mathbf{v}^\top \mathbf{A} \mathbf{v} > 0\) for all \(\mathbf{v} \neq \mathbf{0}\).
• Positive semi-definite (PSD) if \(\mathbf{v}^\top \mathbf{A} \mathbf{v} \geq 0\) for all \(\mathbf{v}\).

Equivalently, \(\mathbf{A}\) is PD iff all its eigenvalues are strictly positive, and PSD iff all eigenvalues are non-negative.

The Gram matrix \(\mathbf{X}^\top \mathbf{X}\) is always PSD, and is PD (hence invertible) when \(\mathbf{X}\) has full column rank — a crucial condition for OLS to have a unique solution.

7 Matrices over a Field and Further Products

For a field \(F\) we write \(M_{n\times m}(F)\) for the \(n\times m\) matrices with entries in \(F\); with entrywise addition and scalar multiplication this is a vector space of dimension \(nm\). If \(A\in M_{n\times m}(F)\) and \(B\in M_{m\times o}(F)\) then \((AB)_{ij}=\sum_{k=1}^ma_{ik}b_{kj}\).

For \(A,B\in M_{n\times m}(F)\) the Hadamard product is the entrywise product \((A\odot B)_{ij}=a_{ij}b_{ij}\). For \(A\in M_{n\times m}(F)\) and \(B\in M_{k\times l}(F)\) the Kronecker product is the \(nk\times ml\) block matrix \[ A\otimes B=\begin{bmatrix}a_{11}B&\cdots&a_{1m}B\\\vdots&\ddots&\vdots\\a_{n1}B&\cdots&a_{nm}B\end{bmatrix}. \]

A square matrix is upper (lower) triangular if all entries below (above) the diagonal are zero.

8 Elementary Operations and Row Reduction

The elementary row operations on \(A\in M_{n\times m}(F)\) are:

1. \(R_i=r_i+\lambda r_j\) for \(\lambda\in F\) and \(j\neq i\) (add a multiple of another row);
2. \(R_i=\lambda r_i\) for \(0\neq\lambda\in F\) (scale a row);
3. \(R_i\leftrightarrow R_j\) (swap rows),

where \(r_i\) denotes the row before the operation and \(R_i\) the row after. Column operations are defined similarly. Matrices related by a sequence of elementary operations are row-equivalent.

A matrix is in reduced row echelon form if

1. every zero row is below every nonzero row;
2. the leading (first nonzero) entry of each nonzero row is \(1\), and these move strictly to the right as we go down the rows;
3. each leading \(1\) is the only nonzero entry in its column.

Every matrix can be put in reduced row echelon form by elementary row operations (Gaussian elimination). Row operations correspond to multiplying on the left by invertible matrices, so they do not change the solution set of \(A\vec{x}=\vec{b}\) (applied to the augmented matrix \([A\mid\vec{b}]\)). A homogeneous system with more unknowns than equations therefore always has a nonzero solution, since a free column remains.

\[ \begin{bmatrix}1&3&3&8&5\\0&1&3&10&8\\0&0&0&-1&-4\\0&0&0&2&8\end{bmatrix} \xrightarrow{\substack{R_3=-r_3\\R_4=r_4+2r_3\text{ (old }r_3)}} \begin{bmatrix}1&3&3&8&5\\0&1&3&10&8\\0&0&0&1&4\\0&0&0&0&0\end{bmatrix} \xrightarrow{\substack{R_1=r_1-3r_2\\R_2=r_2-10r_3}} \begin{bmatrix}1&0&-6&-22&-19\\0&1&3&0&-32\\0&0&0&1&4\\0&0&0&0&0\end{bmatrix} \xrightarrow{R_1=r_1+22r_3} \begin{bmatrix}1&0&-6&0&69\\0&1&3&0&-32\\0&0&0&1&4\\0&0&0&0&0\end{bmatrix}. \]

9 Determinant

The determinant of \(A\in M_n(F)\) is \[ \det(A)=\sum_{\sigma\in S_n}\operatorname{sgn}(\sigma)\prod_{i=1}^na_{i,\sigma(i)}, \] where \(S_n\) is the set of permutations of \(\{1,\dots,n\}\) and \(\operatorname{sgn}(\sigma)=\pm1\) is the sign of \(\sigma\).

For \(n=3\) this gives \(\det\begin{bmatrix}a&b&c\\d&e&f\\g&h&i\end{bmatrix}=a(ei-fh)-b(di-fg)+c(dh-eg)\). Standard properties are \(\det(AB)=\det(A)\det(B)\), \(\det(A^\top)=\det(A)\), and \(\det(A)\neq0\) if and only if \(A\) is invertible. The determinant of a triangular matrix is the product of its diagonal entries.

To find the inverse of an invertible matrix we row reduce the augmented matrix: \([A\mid I_n]\longrightarrow[I_n\mid A^{-1}]\).

10 Trace

The trace of \(A\in M_n(F)\) is the sum of the diagonal entries, \(\operatorname{Tr}(A)=\sum_{i=1}^na_{ii}\).

1. Linearity: \(\operatorname{Tr}(cA+B)=c\operatorname{Tr}(A)+\operatorname{Tr}(B)\).
2. \(\operatorname{Tr}(A^\top)=\operatorname{Tr}(A)\).
3. Cyclicity: \(\operatorname{Tr}(AB)=\operatorname{Tr}(BA)\) whenever both products are defined.
4. For \(A,B\in M_{n\times m}(F)\), \(\operatorname{Tr}(A^\top B)=\sum_{i,j}a_{ij}b_{ij}\).

Proof.

1. and (2) are immediate. (3): \(\operatorname{Tr}(AB)=\sum_i\sum_ka_{ik}b_{ki}=\sum_k\sum_ib_{ki}a_{ik}=\operatorname{Tr}(BA)\). (4): \((A^\top B)_{jj}=\sum_ia_{ij}b_{ij}\), then sum over \(j\).

By cyclicity, similar matrices have equal traces: see Linear Maps.

Backlinks

• Algebra
• Inner Product Space
• Linear Algebra
• Linear Maps
• Matrix Decompositions
• Vector Spaces