1 Optimal Control
In this note we state the deterministic optimal control problem and find the connection between optimal control, Hamilton-Jacobi Equation and the [[hamilton-jacobi-bellman-equation]].
1.1 Optimal Control Problem Formulation
Optimal control problems have system evolution dynamics given by a Markovian ODE \[ \begin{cases} x'(s) = f(x(s), \alpha(s)) & s \in[t,T] \\ x(t)=x, \end{cases} \] where the dynamics apply in time interval \([t,T]\) with \(T\) fixed and an initial value condition given at time \(t\). To see why the dynamics are Markovian we view \(x'(s)\) as \[ x'(s)= \frac{{x(s+h)-x(s)}}{h}\quad h \to 0^+ \] i.e. the ODE is describing the change of state from \(s\) to \(s+h\) given the current state \(x(s)\) and the current control \(\alpha(s)\). The control is understood as the ‘action’ in discrete-time Markov decision process that changes the state evolution and has something to do with rewards.
The control is simply a set of parameters given at each time that will change the dynamics oif the system, eventually changing the state evolution of the system. Let’s denote \(A \subset \mathbb{R}^m\) as some given compact set consisting of all possible values the control at a given time \(\alpha(s)\) can