MTDrive: Multi-turn Interactive Reinforcement Learning for Autonomous Driving

TL;DR

MTDrive tackles long-tail autonomous-driving trajectory planning by enabling iterative, multi-turn reasoning between a Vision-Language Model and environment feedback. It introduces mtGRPO, a per-turn reward and advantage framework $r_{i,j}$ with turn-wise credit assignment, and a data pipeline (single-turn, multi-turn, PDM-understanding) plus a multimodal training system with Inter-Process Streaming Serialization ($IPSS$) and Intra-Process Tensor Cache ($IPTC$). Evaluated on NAVSIM, MTDrive achieves PDMS $=96.2$ with privileged perception and $=91.1$ under a kinematic setting, surpassing several baselines and approaching human performance in certain configurations. The results demonstrate that structured multi-turn reasoning and efficient multimodal training can substantially improve autonomous driving planning, particularly in long-tail scenarios, while reducing data-transfer bottlenecks in multimodal RL.

Abstract

Trajectory planning is a core task in autonomous driving, requiring the prediction of safe and comfortable paths across diverse scenarios. Integrating Multi-modal Large Language Models (MLLMs) with Reinforcement Learning (RL) has shown promise in addressing "long-tail" scenarios. However, existing methods are constrained to single-turn reasoning, limiting their ability to handle complex tasks requiring iterative refinement. To overcome this limitation, we present MTDrive, a multi-turn framework that enables MLLMs to iteratively refine trajectories based on environmental feedback. MTDrive introduces Multi-Turn Group Relative Policy Optimization (mtGRPO), which mitigates reward sparsity by computing relative advantages across turns. We further construct an interactive trajectory understanding dataset from closed-loop simulation to support multi-turn training. Experiments on the NAVSIM benchmark demonstrate superior performance compared to existing methods, validating the effectiveness of our multi-turn reasoning paradigm. Additionally, we implement system-level optimizations to reduce data transfer overhead caused by high-resolution images and multi-turn sequences, achieving 2.5x training throughput. Our data, models, and code will be made available soon.

Figures (4)

• Figure 1: The proposed MTDrive framework. Left: Multi-turn interaction loop---at each turn $i$, the VLM takes front image, ego status, navigation, and historical feedback as input to generate trajectory $\tau_{i+1}$. The Agent evaluates the trajectory and provides per-metric feedback (e.g., collision, drivable area compliance), which is appended for the next turn. Right: Actor update with mtGRPO---unlike standard GRPO which uses a single sequence-level reward, mtGRPO computes per-turn rewards $R_i$ and advantages $A_i$ across multiple rollouts, enabling fine-grained credit assignment for each turn's contribution.
• Figure 2: Overview of the SFT Data Generation Pipeline. Top-left: Single-turn data provides the basic trajectory generation ability which takes front-view image, navigation instruction, historical trajectory (2s), and PDM metric description as input to predict future trajectory (4s). Top-right: PDM understanding data enables the model to interpret PDM feedback through positive/negative QA pairs. Bottom: Multi-turn data is iteratively bootstrapped from single-turn---train on $i$-turn data, run inference, obtain PDM feedback, and stack to form $(i+1)$-turn samples, enabling feedback-guided trajectory refinement.
• Figure 3: Data Transfer Optimization for Multimodal Multi-turn RL Training. (a) Rollout and training workers run in separate processes. IPSS streams serialization during rollout; IPTC enables tensor sharing among co-located modules via a shared cache. (b) IPSS overlaps serialization with rollout generation instead of blocking.
• Figure 4: Multi-turn reasoning visualization. The left figures represent images from the left, center, and right front-facing cameras, while the right figures display the results of one to multi turns of reasoning. In the right images, red crosses indicate trajectory points that violate the DAC metric, solid blue dots indicate obstacles where potential collisions may occur according to the TTC metric, and solid yellow dots indicate obstacles where collisions may occur according to the NC metric.