Lagrange Multiplier Method
1 Intuition
The method of Lagrange multipliers finds the local maxima and minima of a function (such as profit or cost) subject to equality constraints (such as a budget or physical limit). The key idea is that at an optimum the contour lines of the objective are tangent to the constraint surface, so the gradient of the objective is a scalar multiple of the gradient of the constraint.
The method converts a constrained problem into one to which the derivative test of an unconstrained problem can be applied, through the Lagrangian. It is generalised to inequality constraints by the Karush-Kuhn-Tucker conditions (see Optimization).
2 The Lagrangian
For an objective \(f\) and constraint function \(g\) the Lagrangian is \[ \mathcal{L}(x,\lambda):= f(x)+\left< \lambda,g(x) \right>, \] where \(\left< \cdot,\cdot \right>\) is the inner product. The vector \(\lambda\) is the Lagrange multiplier.
To optimise \(f\) subject to \(g(x)=0\) we find the stationary points of \(\mathcal{L}\) in \((x,\lambda)\), \[ \frac{\partial\mathcal{L}}{\partial x}=0\quad \& \quad \frac{\partial\mathcal{L}}{\partial \lambda} = 0, \] or equivalently \[ \frac{\partial f(x)}{\partial x} + \lambda \cdot \frac{\partial g(x)}{\partial x}=0 \quad \& \quad g(x)=0. \] The solution of the constrained problem is a saddle point of the Lagrangian, which can be identified among the stationary points by the definiteness of the bordered Hessian (see Jacobian and Hessian Matrices). The mechanics Lagrangian \(K-U\) is a different object, see Lagrangian Mechanics.
3 Lagrange Multiplier Theorem
Let \(f:\mathbb{R}^n\to\mathbb{R}\) and \(g:\mathbb{R}^n\to\mathbb{R}^c\) be \(C^1\) (continuous first derivatives). Let \(\boldsymbol{x}_{\star}\) maximise \(f(\boldsymbol{x})\) subject to \(g(\boldsymbol{x})=0\), and suppose the matrix of partial derivatives \[ \left[ \mathrm{D} g(\boldsymbol{x}_{\star}) \right] _{j,k}= \frac{\partial g_{j}}{\partial x_{k}} \] has \(\mathrm{rank}(\mathrm{D}g(\boldsymbol{x}_{\star}))= c \leq n\). Then there exists a unique Lagrange multiplier \(\lambda_{\star}\in\mathbb{R}^c\) with \[ \mathrm{D}f(\boldsymbol{x}_{\star})= \lambda_{\star}^{\top}\mathrm{D}g(\boldsymbol{x}_{\star}). \]
In words: at a constrained local extremum, if the constraint qualification (the rank condition) holds, the gradient of the objective is a linear combination of the gradients of the constraints, with the multipliers as coefficients. Equivalently, the directional derivative of \(f\) is zero in every feasible direction: any direction perpendicular to all constraint gradients is also perpendicular to \(\nabla f\).
4 Examples
Find the box of largest volume with total surface area \(64\,\mathrm{cm}^2\). With side lengths \(a,b,c>0\) we maximise \(f(a,b,c)=abc\) subject to \(g(a,b,c)=ab+bc+ac-32=0\). The system is \[ \begin{cases} \nabla f=\lambda\nabla g:\quad \begin{pmatrix} bc \\ ac \\ ab \end{pmatrix} = \lambda \begin{pmatrix} b+c \\ a+c \\ a+b \end{pmatrix}, \\[1ex] g(a,b,c) = 0. \end{cases} \] Multiplying the components of the first equation by \(a,b,c\) respectively gives \(abc=\lambda a(b+c)=\lambda b(a+c)=\lambda c(a+b)\). Since \(abc\neq0\) we have \(\lambda\neq0\), so \(ab+ac=ab+bc=ac+bc\), which forces \(a=b=c\). The constraint then gives \(3a^2=32\), so \(a=b=c=\sqrt{32/3}\): a cube, as expected.
Find the extrema of \(f(x,y,z)=4y-2z\) subject to \(g_{1}=2x-y-z-2=0\) and \(g_{2}=x^2+y^2-1=0\). With two constraints there are two multipliers, and \[ \begin{pmatrix} 0 \\ 4 \\ -2 \end{pmatrix} = \lambda_{1}\begin{pmatrix} 2 \\ -1 \\ -1 \end{pmatrix} +\lambda_{2}\begin{pmatrix} 2x \\ 2y \\ 0 \end{pmatrix},\quad g_{1}=0,\quad g_{2}=0. \] The third component gives \(\lambda_{1}=2\). The first two then give \(x\lambda_{2}=-2\) and \(y\lambda_{2}=3\), and \(g_{2}=0\) yields \(\lambda_{2}=\pm\sqrt{13}\). For \(\lambda_{2}=+\sqrt{13}\), \[ (x,y,z)=\left(-\tfrac{2}{\sqrt{13}},\tfrac{3}{\sqrt{13}}, -2-\tfrac{7}{\sqrt{13}}\right), \] and for \(\lambda_{2}=-\sqrt{13}\), \[ (x,y,z)=\left(\tfrac{2}{\sqrt{13}},-\tfrac{3}{\sqrt{13}}, -2+\tfrac{7}{\sqrt{13}}\right). \] Evaluating \(f\) at the two points identifies the maximum and the minimum.