Hamiltonian
1 What is the Hamiltonian?
In Control Theory the Hamiltonian is a function used to solve for the optimal control of a dynamic system. Specifically, the Hamiltonian is a central object that encodes both the system dynamics and the optimization objective. It can be understood as the instantaneous increment of the Lagrangian expression of the problem that is to be optimized over a certain time period.
2 Hamiltonian Control Problem
We have a system with dynamics given by \(n\) first-order ODEs
\[ \dot{\boldsymbol{x}}(t)=\boldsymbol{f}(\boldsymbol{x}(t), \boldsymbol{u}(t),t), \quad \mathbf{x}(t_0)=\mathbf{x}_0, \]
where:
- \(\boldsymbol{x}(t)=(x_{1}(t), \dots, x_{n}(t))^T\) is the state vector,
- \(\boldsymbol{u}(t):=(u_{1}(t), \dots, u_{r}(t))^T\) is control vector.
We denote by \(\mathcal{X}\subseteq\mathbb{R}^n\) and \(\mathcal{U}\subseteq\mathbb{R}^r\) the sets of admissible states and controls respectively. The solution of the ODEs, given some initial conditions \(\mathbf{x}_0\) and controls \(\mathbf{u}(t)\) is a trajectory \(\boldsymbol{x}(t; \boldsymbol{x}_{0}, t_{0})\).
2.1 Control Problem Formulation
The problem of optimal control is to choose control vector \(\boldsymbol{u}(t)\in\mathcal{U}\) so that \(\boldsymbol{x}(t)\) minimizes (or maximizes) a certain objective over a specified time interval \([t_{0},t_{1}]\). We define the cost functional as
\[ J(\boldsymbol{u})=\int_{t_{0}}^{t_{1}}{L(\boldsymbol{x}(t),\boldsymbol{u}(t),t)}~d{t}+\phi(\boldsymbol{x}(t_{1})), \]
where \(L\) is the running cost (performance index) incurred at each point in time and \(\phi\) is a terminal cost. The problem is then \[ \min_{\boldsymbol{u}(t)\in\mathcal{U}}J(\boldsymbol{u}) \quad\text{subject to}\quad \dot{\boldsymbol{x}}(t)=\boldsymbol{f}(\boldsymbol{x}(t),\boldsymbol{u}(t),t), \quad\boldsymbol{x}(t_{0})=\boldsymbol{x}_{0}. \]
2.2 Solution Method
The solution method involves defining an ancillary function known as the control Hamiltonian
\[ H(\boldsymbol{x}(t), \boldsymbol{u}(t), \boldsymbol{\lambda}(t),t)=L(\boldsymbol{x}(t), \boldsymbol{u}(t), t) + \boldsymbol{\lambda}^T(t)\boldsymbol{f}(\boldsymbol{x}(t), \boldsymbol{u}(t),t), \]
where \(\boldsymbol{\lambda}(t)\in\mathbb{R}^n\) is the costate (or adjoint) vector.
This combines the objective function and the state equations much like a Lagrangian in a static optimization problem only that the multipliers \(\boldsymbol{\lambda}(t)\in\mathbb{R}^n\) - referred to as costate (or adjoint) variables - are functions of time rather than constants.
The Hamiltonian combines the instantaneous cost, via \(L\); and the effect of the control on the future, via \(\boldsymbol{\lambda}^T\boldsymbol{f}\). We want to choose a control \(\boldsymbol{u}(t)\) that not only optimizes the immediate cost, but also drives the system (via \(\boldsymbol{f}\)) in a direction that is favourable in the long run, as reflected by the costate \(\boldsymbol{\lambda}(t)\).
2.3 Necessary Conditions for Optimality
The Pontryagin Maximum Principle states that an optimal pair \((\boldsymbol{x}^*,\boldsymbol{u}^*)\) must satisfy, for some costate \(\boldsymbol{\lambda}(t)\):
- State equation (solved forward in time from \(\boldsymbol{x}(t_{0})\)): \[ \dot{\boldsymbol{x}}(t)=\frac{\partial H}{\partial \boldsymbol{\lambda}}=\boldsymbol{f}(\boldsymbol{x}(t),\boldsymbol{u}(t),t). \]
- Adjoint (costate) equation (solved backward in time from the terminal condition): \[ \dot{\boldsymbol{\lambda}}(t)=-\frac{\partial H}{\partial \boldsymbol{x}}, \qquad \boldsymbol{\lambda}(t_{1})=\frac{\partial \phi}{\partial \boldsymbol{x}}(\boldsymbol{x}(t_{1})). \]
- Optimality condition: the optimal control minimizes the Hamiltonian at each point in time \[ \boldsymbol{u}^*(t)=\arg\min_{\boldsymbol{u}\in\mathcal{U}}H(\boldsymbol{x}^*(t),\boldsymbol{u},\boldsymbol{\lambda}(t),t). \] For an unconstrained interior solution this reduces to \(\partial H/\partial \boldsymbol{u}=\boldsymbol{0}\). For a maximization problem the \(\arg\min\) becomes an \(\arg\max\).
Since \(\boldsymbol{x}\) has a condition at \(t_{0}\) and \(\boldsymbol{\lambda}\) has one at \(t_{1}\), this is a two-point boundary value problem.
3 Worked Example: Linear-Quadratic Regulator (LQR)
Problem Setup:
- System (scalar case): \[ \dot{x}(t)=ax(t)+bu(t). \]
- Cost functional: \[ J(u)=\int_{0}^{T}\left(\tfrac{1}{2}q\,x(t)^2+\tfrac{1}{2}r\,u(t)^2\right)dt, \qquad q\geq 0,~r>0. \]
We find the optimal control \(u^*(t)\) that minimizes \(J(u)\).
3.1 Step 1: Define the Hamiltonian
With scalar costate \(\lambda(t)\in\mathbb{R}\), \[ H(x,u,\lambda,t)=\tfrac{1}{2}q x^2+\tfrac{1}{2}r u^2+\lambda(ax+bu). \]
3.2 Step 2: Find the optimal control
We minimize \(H\) with respect to \(u\): \[ \frac{\partial H}{\partial u}=ru+\lambda b=0 \quad\Rightarrow\quad u^*(t)=-\frac{b}{r}\lambda(t). \]
3.3 Step 3: State and adjoint equations
- State: \[ \dot{x}(t)=ax(t)+bu^*(t)=ax(t)-\frac{b^2}{r}\lambda(t). \]
- Adjoint: \[ \dot{\lambda}(t)=-\frac{\partial H}{\partial x}=-qx(t)-a\lambda(t), \qquad \lambda(T)=0. \]
3.4 Step 4: Solve the system
This gives a coupled system of ODEs for \(x(t)\) and \(\lambda(t)\). Posing \(\lambda(t)=P(t)x(t)\) and substituting into the adjoint equation yields the Riccati equation \[ -\dot{P}(t)=2aP(t)-\frac{b^2}{r}P(t)^2+q,\qquad P(T)=0, \] and the optimal control takes the feedback form \(u^*(t)=-\frac{b}{r}P(t)x(t)\).
4 Summary
- The Hamiltonian in control theory is a function combining cost and dynamics.
- It is used to formulate necessary conditions for optimality via the Pontryagin Maximum Principle.
- The optimal control minimizes the Hamiltonian, with the state and costate evolving according to the system and adjoint dynamics.
- In problems like LQR, it leads to elegant analytic solutions.
5 Linked Topics:
- Pontryagin Maximum Principle