Optimal Feedback Control of Astrodynamic Systems Using Solutions of the Hamilton-Jacobi-Bellman Equation

Optimal control of dynamic systems in the presence of model and initial condition errors and perturbations in flight is important but difficult to implement. An obvious approach would be to solve for the open loop optimal trajectory from the off-nominal state. However this is impractical in real time for anything other than simple dynamic systems. A common approach is to use the necessary conditions at second order, the so-called accessory minimum problem, to derive an approximate, neighboring optimal guidance (NOG) law. The most satisfactory solution, if it can be achieved, is to solve the system Hamilton-Jacobi-Bellman (HJB) partial differential equation for the value function at all points within the likely domain. With this knowledge it is possible to determine the exact optimal control required at any off-nominal state. However, the space complexity of the problem is exponential with respect to the number of dimensions of the system. Moreover, the value function of the HJB equation may be nondifferentiable, which renders traditional PDE solution methods impractical. Therefore, extant methods are suitable only for special problem classes such as those involving affine systems or where the value function is differentiable. This chapter will describe recent advances toward making such solutions tractable, including PDE viscosity solutions, quasi-Monte Carlo grids, Kriging regression, and automatic mesh refinement (AMR) using reinforcement learning (RL). Examples are provided for several problems relevant to astrodynamics.