Jacobian and Hessian Matrices
1 Jacobian Matrix
In Vector Calculus the Jacobian matrix of a vector function \(\boldsymbol{f}:\mathbb{R}^n\to\mathbb{R}^m\) collects all of its first-order partial derivatives. It is the natural generalisation of the derivative to vector-valued functions. If the matrix is square (\(n=m\)) its determinant is the Jacobian determinant.
Let \(\boldsymbol{f}:\mathbb{R}^n\to\mathbb{R}^m\) be a vector function whose first-order partial derivatives all exist. The Jacobian matrix \(J_{\boldsymbol{f}}\) is the \(m\times n\) matrix with \((i,j)\)-th entry \(\frac{\partial f_{i}}{\partial x_{j}}\):
\[ J_{\boldsymbol{f}}=\begin{bmatrix} \frac{\partial \boldsymbol{f}}{\partial x_{1}} & \cdots & \frac{\partial \boldsymbol{f}}{\partial x_{n}} \end{bmatrix}=\begin{bmatrix} \nabla^{\top}f_{1} \\ \vdots \\ \nabla^{\top}f_{m} \end{bmatrix}=\begin{bmatrix} \frac{\partial f_{1}}{\partial x_{1}} & \cdots & \frac{\partial f_{1}}{\partial x_{n}} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_{m}}{\partial x_{1}} & \cdots & \frac{\partial f_{m}}{\partial x_{n}} \end{bmatrix}, \]
where \(\nabla^{\top}f_{i}\) is the row vector given by the transpose of the Gradient of the \(i\)-th component (see Divergence).
Let \(f(x,y)=(2x^2+3xy,\ 4y^2x+2x)\). Then \[ J_{f}=\begin{bmatrix} 4x+3y & 3x \\ 4y^2+2 & 8xy \end{bmatrix}, \] and the Jacobian determinant is \[ \det(J_{f})=8xy(4x+3y)-3x(4y^2+2)=32x^2y+12xy^2-6x. \] The Jacobian describes the change in the new coordinates with respect to the old coordinates.
1.1 Best Linear Approximation
If \(\boldsymbol{f}\) is differentiable at \(\boldsymbol{x}^*\in\mathbb{R}^n\) then its differential is represented by \(J_{\boldsymbol{f}}(\boldsymbol{x}^*)\), and the linear map \(\boldsymbol{h}\mapsto J_{\boldsymbol{f}}(\boldsymbol{x}^*) \boldsymbol{h}\) is the best linear approximation of \(\boldsymbol{f}\) near \(\boldsymbol{x}^*\):
\[ \boldsymbol{f}(\boldsymbol{x})-\boldsymbol{f}(\boldsymbol{x}^*)= J_{\boldsymbol{f}}(\boldsymbol{x}^*)(\boldsymbol{x}-\boldsymbol{x}^*) +o(\lVert\boldsymbol{x}-\boldsymbol{x}^*\rVert),\qquad \boldsymbol{x}\to\boldsymbol{x}^*, \]
where \(o(\lVert\boldsymbol{x}-\boldsymbol{x}^*\rVert)\) is a remainder that vanishes faster than \(\lVert\boldsymbol{x}-\boldsymbol{x}^*\rVert\). For \(n=m=1\) this is the degree-one Taylor Series approximation \(f(x)-f(x^*)=f'(x^*)(x-x^*)+o(x-x^*)\), so the Jacobian is the first-order derivative of a vector function.
1.2 Chain Rule
For composable differentiable functions \(f:\mathbb{R}^n\to\mathbb{R}^m\) and \(g:\mathbb{R}^m\to\mathbb{R}^k\), and \(x\in\mathbb{R}^n\), \[ J_{g\circ f}(x)=J_{g}(f(x))\,J_{f}(x). \]
Proof. By the best linear approximation of \(f\) at \(x\) and of \(g\) at \(f(x)\), with \(h\to0\), \[ g(f(x+h))=g\bigl(f(x)+J_{f}(x)h+o(\lVert h\rVert)\bigr) =g(f(x))+J_{g}(f(x))J_{f}(x)h+o(\lVert h\rVert). \] The matrix multiplying \(h\) is the Jacobian of \(g\circ f\) at \(x\). This is the matrix form of the Chain Rule.
1.3 Inverse
By the inverse function theorem, if \(f:\mathbb{R}^n\to\mathbb{R}^n\) has a continuous Jacobian that is non-singular at \(x\), then \(f\) is invertible on a neighbourhood of \(x\), i.e. it is locally invertible. The Jacobian of the inverse is the matrix inverse of the Jacobian, \[ J_{f^{-1}}(y)=J_{f}(f^{-1}(y))^{-1},\qquad \det J_{f^{-1}}(y)=\frac{1}{\det J_{f}(f^{-1}(y))}. \] This follows from the chain rule applied to \(f\circ f^{-1}=\mathrm{id}\). These formulas drive the Change-of-Variables Formula.
1.4 Critical Points
For a differentiable \(f:\mathbb{R}^n\to\mathbb{R}^m\), a critical point is a point where the rank of \(J_{f}\) is not maximal, i.e. is smaller than \(\min(n,m)\). For \(m=1\) these are the points where \(\nabla f=0\); for \(n=m\) they are the points where \(\det J_{f}=0\).
2 Hessian Matrix
The Hessian matrix is the second-order analogue of the Jacobian, for a scalar field. It describes the local curvature of a multivariate function. It is usually denoted \(H\) (historically also \(\nabla\nabla\), \(\nabla^2\), \(\nabla\otimes\nabla\) or \(D^2\)).
Let \(f:\mathbb{R}^n\to\mathbb{R}\) have all second-order partial derivatives. The Hessian matrix of \(f\) is the \(n\times n\) matrix \[ (H_{f})_{i,j}:=\frac{\partial^2 f}{\partial x_{i}\partial x_{j}}. \] Its determinant is the Hessian determinant.
If the second partial derivatives are continuous, then \(H_{f}\) is a symmetric matrix by the symmetry of mixed partial derivatives. The Hessian is the Jacobian of the gradient, transposed: \[ H_{f}(x)=J_{\nabla f}(x)^{\top}. \] (The two coincide when \(H_f\) is symmetric.)
For \(f(x,y)=x^2y+y^3\), \[ \nabla f=\begin{pmatrix}2xy\\ x^2+3y^2\end{pmatrix},\qquad H_{f}=\begin{bmatrix}2y & 2x\\ 2x & 6y\end{bmatrix}, \] which is symmetric, as expected.
2.1 Applications to Optimisation
The Hessian is the coefficient of the quadratic term in the second-order Taylor expansion \[ f(\boldsymbol{x}+\Delta\boldsymbol{x})\approx f(\boldsymbol{x}) +\nabla f(\boldsymbol{x})^{\top}\Delta\boldsymbol{x} +\tfrac{1}{2}\Delta\boldsymbol{x}^{\top}H_{f}(\boldsymbol{x})\Delta\boldsymbol{x}. \] At a critical point (\(\nabla f=0\)) the sign of the quadratic form decides the type of the point: a minimum if \(H_f\) is positive definite, a maximum if it is negative definite and a saddle if it is indefinite (see positive definite matrices). Newton-type methods in Optimization use this expansion to choose their steps.
Computing and storing the full Hessian takes \(\mathcal{O}(n^2)\) memory, which is infeasible for high-dimensional problems such as the loss functions of neural networks or statistical models with many parameters.