Control Theory
1 Intuition
In control theory we steer a dynamical system by choosing an input (the control) over time so as to optimize a payoff. It is an individual decision problem: there are no competing players, which distinguishes it from Game Theory. When the dynamics contain randomness the payoff is an expectation and we are in Stochastic Control.
The central idea is that we optimize over a function (the control path), an infinite dimensional problem, which is why the tools come from the Calculus of Variations (Euler-Lagrange Equation, Hamilton’s equations, Hamilton-Jacobi Equation).
2 Problem formulation
Let \(A\subset\mathbb{R}^m\) be a compact set of admissible control values. A control is a (measurable) function \(\alpha:[0,T]\to A\). Given \(f:\mathbb{R}^n\times A\to\mathbb{R}^n\) (written \(f(x,a,t)\) if it depends on time), the controlled dynamical system is \[ \begin{cases} \dot{x}(s)=f(x(s),\alpha(s)), & s\in[t,T],\\ x(t)=x, \end{cases} \] started at time \(t\) from state \(x\in\mathbb{R}^n\). It is autonomous when \(f\) has no explicit time dependence.
The dynamics are Markovian: the change of state from \(s\) to \(s+h\) depends only on the current state \(x(s)\) and current control \(\alpha(s)\), as in the action of a discrete-time Markov decision process (see Markov Chains).
The payoff (a functional of the control) is \[ P_{t,x}[\alpha]=\int_{t}^{T}r(x(s),\alpha(s))\,ds+g(x(T)), \] where \(r\) is the running payoff and \(g\) the terminal payoff.
The optimal control problem is to find \(\alpha^*\) with \(P_{t,x}[\alpha^*]\geq P_{t,x}[\alpha]\) for every admissible \(\alpha\). In practice we require \(f\), \(r\), \(g\) to be regular, e.g. Lipschitz continuous, so that the ODE is well posed.
An individual starts at the origin of a line and is rewarded for being close to it after one minute, while moving at speed at most 1. Then \(\dot{x}=\alpha\), \(x(0)=0\), \(A=[-1,1]\), \(r=0\), \(g(x)=-|x|\), so \(P[\alpha]=-|x(1)|\) is to be maximized. The optimal strategy is \(\alpha\equiv0\).
3 Value function and dynamic programming
The value function is the best attainable payoff from state \(x\) at time \(t\): \[ u(x,t)=\sup_{\alpha}P_{t,x}[\alpha],\qquad u(x,T)=g(x). \]
The value function obeys the Dynamic Programming Principle: for \(t\le t+h\le T\), \[ u(x,t)=\sup_{\alpha}\left\{\int_{t}^{t+h}r(x(s),\alpha(s))\,ds +u(x(t+h),t+h)\right\}. \] Letting \(h\to0\) formally yields the first-order PDE \[ -\partial_{t}u(x,t)=\sup_{a\in A}\left\{f(x,a)\cdot\nabla u(x,t)+r(x,a)\right\}, \] a Hamilton-Jacobi Equation with Hamiltonian \(H(x,p)=\sup_{a}\{f(x,a)\cdot p+r(x,a)\}\) (see Hamiltonian Mechanics, Gradient). In the stochastic case a second-order term appears, giving the Hamilton-Jacobi-Bellman Equation.
4 Methods
- Calculus of variations / Pontryagin: necessary conditions for an optimal path via Hamilton’s equations (see Hamiltonian Mechanics) and Lagrange Multiplier Method for the dynamic constraint.
- Dynamic programming: solve the Hamilton-Jacobi(-Bellman) equation for \(u\), then choose \(\alpha^*(s)\) as a maximizer of the Hamiltonian along the optimal trajectory.
5 References
- L. C. Evans, Partial Differential Equations, Chapter 10.
- W. H. Fleming and R. W. Rishel, Deterministic and Stochastic Optimal Control.