Learning the optimal state-feedback via supervised imitation learning

TL;DR

This work investigates learning near-optimal state-feedback maps for deterministic optimal control by imitating optimal trajectories of a 2D quadcopter. It generates a large dataset via direct trajectory optimization and trains deep neural networks to map states to controls under two objectives, $J=\int_0^T (F_T^2+\omega^2)\,dt$ (QOC) and $J=T$ (TOC), evaluating both trajectory accuracy and asymptotic behavior. The study finds that, with 2–6 hidden layers, the learned policies achieve sub-one-percent deviations from optimal costs, and that a softplus activation yields smoother controls with comparable or improved performance over ReLU networks. These results demonstrate a scalable imitation-learning pipeline for precise state-feedback in low-dimensional systems and motivate extending the approach to higher-dimensional dynamics.

Abstract

Imitation learning is a control design paradigm that seeks to learn a control policy reproducing demonstrations from expert agents. By substituting expert demonstrations for optimal behaviours, the same paradigm leads to the design of control policies closely approximating the optimal state-feedback. This approach requires training a machine learning algorithm (in our case deep neural networks) directly on state-control pairs originating from optimal trajectories. We have shown in previous work that, when restricted to low-dimensional state and control spaces, this approach is very successful in several deterministic, non-linear problems in continuous-time. In this work, we refine our previous studies using as a test case a simple quadcopter model with quadratic and time-optimal objective functions. We describe in detail the best learning pipeline we have developed, that is able to approximate via deep neural networks the state-feedback map to a very high accuracy. We introduce the use of the softplus activation function in the hidden units of neural networks showing that it results in a smoother control profile whilst retaining the benefits of rectifiers. We show how to evaluate the optimality of the trained state-feedback, and find that already with two layers the objective function reached and its optimal value differ by less than one percent. We later consider also an additional metric linked to the system asymptotic behaviour - time taken to converge to the policy's fixed point. With respect to these metrics, we show that improvements in the mean absolute error do not necessarily correspond to better policies.

Figures (4)

• Figure 1: Two-dimensional model of a quadcopter considered in this study. Control inputs $\{F_T, \omega\}$ as well as the state variables $\{x, z, \theta\}$ are indicated.
• Figure 2: Example state-control trajectory for the quadcopter model with quadratic (QOC) and regularised time (TOC) objective functions. For the presented example, the initial state is ($x=-5, z=0, v_x=2, v_z=2, \theta=0$) and the target state is $\mathbf{0}$ (green dot). (a) State trajectory plotted for the position variables $(x,z)$ with the pitch angle indicated. (b) Corresponding optimal control profiles. We note that with respect to the thrust profile (top right), QOC is continuous and TOC is bang-bang.
• Figure 3: Learning rate finder method from smith2017cyclical. Learning rate is increased on a logarithmic scale during training of a single epoch starting from a small value. Optimal learning rate is the one that gives the largest decrease in the loss. Left: Initially with small learning rates, the loss improves slowly. As the learning rate increases, the loss improves faster until it explodes (not shown) when the learning rate becomes very large. Fastest decrease in the loss is when the slope of the loss curve is the most negative (dotted red line). Right: Plot of the change in loss per iteration with a moving average filter applied. The optimal learning rate is the minimum of this curve (dotted red line).
• Figure 4: Comparison of the control profiles of policies derived from ReLU and softplus neural networks for quadratic optimal control. The controls necessary for the system to be at equilibrium are indicated (grey dashed line).