End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

TL;DR

The paper presents Roach, an RL coach that maps BEV inputs to continuous low-level driving actions and provides rich supervision signals for imitation learning. Roach uses a Beta-distributed action head and PPO with an exploration loss to achieve a new CARLA upper bound and robust, sample-efficient training across six maps. By supervising IL agents with Roach’s action distributions, latent features, and value estimates (L_K, L_F, L_V), the authors achieve expert-level, single-camera end-to-end driving and state-of-the-art results on the CARLA LeaderBoard, with strong generalization to new towns and weather. This approach significantly improves sample efficiency and generalization for end-to-end urban driving, suggesting practical potential for real-world deployment and scalable on-policy supervision.

Abstract

End-to-end approaches to autonomous driving commonly rely on expert demonstrations. Although humans are good drivers, they are not good coaches for end-to-end algorithms that demand dense on-policy supervision. On the contrary, automated experts that leverage privileged information can efficiently generate large scale on-policy and off-policy demonstrations. However, existing automated experts for urban driving make heavy use of hand-crafted rules and perform suboptimally even on driving simulators, where ground-truth information is available. To address these issues, we train a reinforcement learning expert that maps bird's-eye view images to continuous low-level actions. While setting a new performance upper-bound on CARLA, our expert is also a better coach that provides informative supervision signals for imitation learning agents to learn from. Supervised by our reinforcement learning coach, a baseline end-to-end agent with monocular camera-input achieves expert-level performance. Our end-to-end agent achieves a 78% success rate while generalizing to a new town and new weather on the NoCrash-dense benchmark and state-of-the-art performance on the challenging public routes of the CARLA LeaderBoard.

Figures (9)

• Figure 1: Roach: RL coach allows IL agents to benefit from dense and informative on-policy supervisions.
• Figure 2: The BEV representation used by our Roach.
• Figure 3: Network architecture of Roach, the RL expert, and CILRS, the IL agent.
• Figure 4: Learning curves of RL experts trained in CARLA Town 1-6. Solid lines show the mean and shaded areas show the standard deviation of episode returns across 3 seeds. The dashed line shows an outlier run that collapsed.
• Figure 5: Driving score of experts and IL agents. All IL agents (dashed lines) are supervised by Roach except for $\mathcal{L}_\text{A}(\text{AP})$, which is supervised by our Autopilot. For IL agents at the 5th iteration on NoCrash and all experts, results are reported as the mean over 3 evaluation seeds. Others are evaluated with one seed. The offline Leaderboard benchmark is used here.