Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environments

TL;DR

A hierarchical GAIL-based architecture (hGAIL) which decouples representation learning from the driving task to solve the autonomous navigation of a vehicle and is able to learn both the policy and the mid-level representation simultaneously as the agent interacts with the environment.

Abstract

Deriving robust control policies for realistic urban navigation scenarios is not a trivial task. In an end-to-end approach, these policies must map high-dimensional images from the vehicle's cameras to low-level actions such as steering and throttle. While pure Reinforcement Learning (RL) approaches are based exclusively on engineered rewards, Generative Adversarial Imitation Learning (GAIL) agents learn from expert demonstrations while interacting with the environment, which favors GAIL on tasks for which a reward signal is difficult to derive, such as autonomous driving. However, training deep networks directly from raw images on RL tasks is known to be unstable and troublesome. To deal with that, this work proposes a hierarchical GAIL-based architecture (hGAIL) which decouples representation learning from the driving task to solve the autonomous navigation of a vehicle. The proposed architecture consists of two modules: a GAN (Generative Adversarial Net) which generates an abstract mid-level input representation, which is the Bird's-Eye View (BEV) from the surroundings of the vehicle; and the GAIL which learns to control the vehicle based on the BEV predictions from the GAN as input. hGAIL is able to learn both the policy and the mid-level representation simultaneously as the agent interacts with the environment. Our experiments made in the CARLA simulation environment have shown that GAIL exclusively from cameras (without BEV) fails to even learn the task, while hGAIL, after training exclusively on one city, was able to autonomously navigate successfully in 98% of the intersections of a new city not used in training phase. Videos and code available at: https://sites.google.com/view/hgail

Figures (13)

• Figure 1: a) Images from three frontal cameras located at the left, central, and right part of the vehicle, respectively. They were taken after the first few interactions of the agent in the CARLA simulation environment. Each camera produces a 256x144 RGB image. b) The corresponding sparse trajectory visual input captured at the same frame. The points from the sparse trajectory and the highlighted vehicle position are plotted as circles with a radius of 10 pixels, using the same scale (pixels per meter) and perspective as the BEV representation. When the image is fed to the CGAN, it is represented with only one channel and a size of 192x192 pixels. c) The three channels of BEV image that our agent employs, computed at the same instant shown in (a). From left to right, the channels correspond to: desired route, drivable area, and lane boundaries. The last image shows all three channels combined in different colors.
• Figure 2: Hierarchical Generative Adversarial Imitation Learning (hGAIL) for policy learning with mid-level input representation. It basically consists of chained CGAN and GAIL networks, where the first one (CGAN) generates BEV representation from the vehicle's three frontal cameras, sparse trajectory and high-level command, while the latter (GAIL) outputs the acceleration and steering based on the predicted BEV input (generated by CGAN), the current speed and the last applied actions. Both CGAN and GAIL learn simultaneously while the agent interacts to the CARLA environment. The discriminator parts of both networks are not shown for the sake of simplicity.
• Figure 3: Town01 environment of the agent, with one of the routes used to collect data by the expert. The highlighted path has 740 meters, 20 points in the sparse trajectory (shown as yellow dots) and 762 points in the dense point trajectory (not shown).
• Figure 4: Number of committed infractions vs. environment interactions during training in town1 environment. The top (bottom) plot shows the results for the agents receiving (disregarding) the sparse trajectory as visual input (trajectory 1x192x192 in Fig.\ref{['fig:gan']}) in the corresponding CNN. For each method (hGAIL, GAIL with real Bird's-Eye View, GAIL from cameras), the average performance of three runs is depicted considering a stochastic policy. The shaded area represents the standard deviation. The GAIL from cameras agent fails to learn the task and keep the sum of committed infractions close to zero, while the goal of zero infractions is achieved by both hGAIL and GAIL with real BEV.
• Figure 5: Evaluation of agents in town2, trained exclusively in town1. The plot shows the percentage of completed routes from a total of six Leaderboard routes in town2 vs. environment interactions, averaged over three different runs, where each run entails a different agent trained only in town1. For each method (hGAIL, GAIL with real Bird's-Eye View, hGAIL ablated), the average performance of three runs is depicted considering a deterministic policy. The shaded area represents the standard deviation. Not shown in the plot, Behavior Cloning (BC) and GAIL from cameras agents fail to learn the task and complete any route (staying at 0% if shown in the plot). Both hGAIL and GAIL with real BEV agents are able to generalize the learning in town1 to town2. The latter agent does not have to learn BEV, as it has always access to the true BEV. The hGAIL ablated agent receives no visual input of the sparse trajectory, but only its numeric vector.