1 Hessian Matrix
1.1 Hessian Matrix
The Hessian matrix is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field. The Hessian describes the local curvature of a multivariate function. The Hessian is typically denoted \(H\) but has historically also been denoted by \(\nabla \nabla\) or \(\nabla ^2\) or \(\nabla \otimes \nabla\) or \(D^2\).
Consider the scalar function \(f:\mathbb{R}^n\to \mathbb{R}\). If all second-order partial derivatives of \(f\) exist, then the Hessian matrix \(H\) of \(f\) is a square \(n \times n\) matrix defined as \[ H_{f}=(H_{f})_{i,j}:=\left(\frac{{\partial^2f}}{\partial x_{i}\partial x_{j}}\right)_{i,j}. \]
Further, if the second partial derivatives are all continuous, the Hessian matrix is a symmetric matrix by the symmetry of second derivatives. The determinant of the Hessian matrix is called the Hessian determinant.
The Hessian matrix of a function \(F\) is the transpose of the Jacobian Matrix of the Gradient of the function \(f\). That is \[ H(f(x))=J(\nabla f(x))^\top. \] ## Applications
1.1.1 Optimization
References: Wikipedia.
Hessian matrices are used in large-scale optimization problems with Newton-type methods because they are the coefficient of the quadratic term of a local Taylor expansion of a function. That is \[ y = f(\boldsymbol{x}+ \Delta \boldsymbol{ x})\approx f(\boldsymbol{x})+\nabla f(\boldsymbol{x})^\top \Delta \boldsymbol{x}+ \frac{1}{2}\Delta \boldsymbol{x}^\top H(\boldsymbol{x})\Delta \boldsymbol{x}, \] where \(\nabla f\) is the Gradient.
Computing and storing the full Hessian takes \(\mathcal{O}(n^2)\) memory, which is infeasible for high-dimensional functions such as the loss functions of Neural Networks, Conditional random fields and other statistical models with large numbers of parameters.