CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving

TL;DR

CoIRL-AD addresses the generalization gap of imitation learning and the sample inefficiency of reinforcement learning in end-to-end autonomous driving by introducing a dual-policy framework that jointly trains IL and RL agents through a shared latent world model. It employs imagination-based RL with group sampling, a backward planning scheme via inverse causality, and a competition-driven mechanism that transfers knowledge between actors without gradient conflicts. Key contributions include integrating RL into an offline, latent-space driving framework, a step-aware RL strategy with a dreaming critic, and a flexible decision rule for IL/RL interaction, validated on nuScenes and Navsim with reduced collisions and improved long-tail performance. The approach offers a practical path to more robust, generalizable autonomous driving systems leveraging both imitation and reinforcement signals with minimal external simulators.

Abstract

End-to-end autonomous driving models trained solely with imitation learning (IL) often suffer from poor generalization. In contrast, reinforcement learning (RL) promotes exploration through reward maximization but faces challenges such as sample inefficiency and unstable convergence. A natural solution is to combine IL and RL. Moving beyond the conventional two-stage paradigm (IL pretraining followed by RL fine-tuning), we propose CoIRL-AD, a competitive dual-policy framework that enables IL and RL agents to interact during training. CoIRL-AD introduces a competition-based mechanism that facilitates knowledge exchange while preventing gradient conflicts. Experiments on the nuScenes dataset show an 18% reduction in collision rate compared to baselines, along with stronger generalization and improved performance on long-tail scenarios. Code is available at: https://github.com/SEU-zxj/CoIRL-AD.

Figures (17)

• Figure 1: Overview of CoIRL-AD. CoIRL-AD adopts a dual-policy architecture that integrates imitation learning (IL) and reinforcement learning (RL) through a shared latent world model. In each iteration, the IL actor and RL actor are trained in parallel. The latent world model is learned during the IL phase and then used in the RL phase, where only the RL actor and critic are updated. For exploration, the RL actor samples multiple action sequences, predicts future states via the latent world model, and evaluates them with rule-based reward functions. The critic assigns advantages to each sequence based on the imagined future states and rewards. To promote interaction, a competitive learning mechanism exchanges knowledge between the IL and RL actors.
• Figure 2: The comparison of forward planning (using causal mask) and backward attention (use inverse causal mask)
• Figure 3: The flow chart of our rule-based competitive learning mechanism
• Figure 4: Comparision of Generalization and Performance on Long-tail Dataset (L2 and Collision).
• Figure 5: Accumulated wins (top) and score difference (bottom) across training iterations.