Inverse RL Scene Dynamics Learning for Nonlinear Predictive Control in Autonomous Vehicles

TL;DR

The paper addresses autonomous vehicle control under dynamic driving scenes by coupling a nominal, physics-based vehicle model with a learned scene-dynamics model. The scene dynamics are encoded in a deep network trained via an inverse reinforcement learning framework, leveraging an Augmented Memory to fuse past observations into trajectory predictions for a constrained nonlinear MPC. Across GridSim, indoor/outdoor AMTU tests, and a full-scale VW Passat on public roads, the approach outperforms Dynamic Window, End2End, and DQN baselines in safety and tracking metrics, particularly in obstacle-rich environments. The work demonstrates a principled hybrid of model-based control and environment-aware learning, with discussions on safety, data needs, and avenues for future stability and terminal-constraint enhancements.

Abstract

This paper introduces the Deep Learning-based Nonlinear Model Predictive Controller with Scene Dynamics (DL-NMPC-SD) method for autonomous navigation. DL-NMPC-SD uses an a-priori nominal vehicle model in combination with a scene dynamics model learned from temporal range sensing information. The scene dynamics model is responsible for estimating the desired vehicle trajectory, as well as to adjust the true system model used by the underlying model predictive controller. We propose to encode the scene dynamics model within the layers of a deep neural network, which acts as a nonlinear approximator for the high order state-space of the operating conditions. The model is learned based on temporal sequences of range sensing observations and system states, both integrated by an Augmented Memory component. We use Inverse Reinforcement Learning and the Bellman optimality principle to train our learning controller with a modified version of the Deep Q-Learning algorithm, enabling us to estimate the desired state trajectory as an optimal action-value function. We have evaluated DL-NMPC-SD against the baseline Dynamic Window Approach (DWA), as well as against two state-of-the-art End2End and reinforcement learning methods, respectively. The performance has been measured in three experiments: i) in our GridSim virtual environment, ii) on indoor and outdoor navigation tasks using our RovisLab AMTU (Autonomous Mobile Test Unit) platform and iii) on a full scale autonomous test vehicle driving on public roads.

Figures (14)

• Figure 1: Autonomous driving problem space in the GridSim simulator Trasnea_IRC2019. Given current and past vehicle states $\mathbf{z}^{<t-\tau_i, t>}$, a reference trajectory $\mathbf{z}_{ref}^{<t-\infty, t+\infty>}$ and an input sequence of observations $\mathbf{I}^{<t-\tau_i, t>}$, the goal is to estimate the vehicle's future state trajectory $\mathbf{z}^{<t+1, t+\tau_o>}$ and optimal control actions $\mathbf{u}^{<t+1, t+\tau_o>}$, over control horizon $\tau_o$.
• Figure 2: DL-NMPC-SD: Deep Learning-based Nonlinear Model Predictive Controller using Scene Dynamics for autonomous navigation. The desired trajectory $\mathbf{z}^{<t+1, t+\tau_o>}$ for the constrained NMPC is estimated by the learned scene dynamics model $\mathbf{h}(\cdot)$ added to the nominal process model, based on historical sequences $\mathbf{s}^{<t>} = (\mathbf{z}^{<t-\tau_i, t>}, \mathbf{I}^{<t-\tau_i, t>})$ of system states $\mathbf{z}^{<t-\tau_i, t>}$ and observations $\mathbf{I}^{<t-\tau_i, t>}$, each integrated by the Augmented Memory component. The input required by the controller is the global start-to-destination route $\mathbf{z}^{<t-\infty, t+\infty>}_{ref}$ for the vehicle, which, from a control perspective, varies in the $(t-\infty, t+\infty)$ interval. $\tau_i$ and $\tau_o$ represent past (input) and future (output) temporal horizons, respectively. The dotted lines illustrate the flow of data used during training.
• Figure 3: Examples of synthetic GridSim (a,b) and real-world (c,d) occupancy grids. The images in each group show visual snapshots of the driving environment together with their respective OGs and activations of the first convolutional layer of the network from Fig. \ref{['fig:neural_network_diagram']}.
• Figure 4: A folded (a) and unfolded (b) over time, many-to-many Recurrent Neural Network. Both the input $\mathbf{s}^{<t-\tau_i, t>}$ and output $\mathbf{z}^{<t+1, t+\tau_o>}_d$ sequences share the same weights over time $t$.
• Figure 5: Deep Neural Network architecture for encoding the scene dynamics model $\mathbf{h}(\cdot)$. Combined with the nominal process model, the output of the network is used to calculate desired policies along receding horizon $\mathbf{z}^{<t+1, t+\tau_o>}_d$. The training data consists of synthetic and real-world observation sequences $\mathbf{I}^{<t-\tau_i, t>}$, vehicle states $\mathbf{z}^{<t-\tau_i, t>}$, and reference trajectories $\mathbf{z}^{<t-\tau_i, t+\tau_o>}_{ref}$, together with their respective desired states $\mathbf{z}^{<t+1, t+\tau_o>}_d$. Before being stacked upon each other, the observations stream is passed through a convolutional neural network, having two convolutional blocks of $9.472$ and $896$ units, respectively. The input vehicle and reference states are firstly process by LSTM modules, each composed of $64$ hidden units. The merged data is further fed to separate LSTMs via two fully connected layers of $1536$ and $1024$ units, respectively. Each LSTM branch is responsible for predicting a compensating factor $\mathbf{h}^{<t+i>}$, where $i$ is an index in the receding horizon.