Generative Scenario Rollouts for End-to-End Autonomous Driving

TL;DR

GeRo addresses the limitations of existing Vision-Language-Action driving models by introducing a generative scenario rollout framework that jointly learns planning and autoregressive, language-conditioned future traffic scenes. It first pretrains a VLA model to encode ego and agent dynamics into a latent token space and then performs language-guided rollout to generate future tokens, trajectories, and egocentric language outputs over long horizons. Rollout consistency losses and GRPO-based reinforcement learning stabilize predictions and align language with actions, yielding safer, more interpretable planning. On Bench2Drive and nuScenes, GeRo delivers state-of-the-art performance across closed-loop and open-loop metrics and demonstrates strong zero-shot generalization, underscoring the value of language-grounded scenario reasoning for robust autonomous driving.

Abstract

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and under-utilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.

Figures (4)

• Figure 1: (Top) Closed-loop metrics (Driving Score and Success Rate) and open-loop metric (Trajectory L2 error) for different models. GeRo consistently improves both closed-loop and open-loop performance across baselines like Qwen2.5VL Qwen2.5-VL and ORION fu2025orion. (Bottom) GeRo scenario rollout pipeline: given a scenario description and multi-view observations, GeRo generates temporally consistent agent behaviors using a large language model (LLM) across time steps.
• Figure 2: An overview of GeRo's two stage framework. Left: GeRo pretraining stage of the Vision–Language–Action (VLA) model. Multi-view images and scenario prompts are encoded into visual and text tokens and projected by the LLM head to compute ego and agent tokens. These tokens drive are passed to the generative planner and the motion prediction head. The planning, motion and language prediction tasks are supervised by their respective loss functions($\mathcal{L}_{\text{plan}}, \mathcal{L}_{\text{mot}}, \mathcal{L}_{\text{VLA}}$). . Right: GeRo's autoregressive rollout stage. Given a scenario description and ego-action questions, GeRo performs scenario rollout to predict future tokens for $T$ steps, decoding them into ego waypoints, multi-agent trajectories, and language outputs. Rollouts are optimized with consistency loss ($\mathcal{L}_{\text{roll}}$) and reinforcement feedback ($\mathcal{L}_{GRPO}$) for robust scenario grounding.
• Figure 3: Autoregressive scenario rollout in GeRo. At time $t$, GeRo consumes ego and agent tokens along with scenario description and ego-action questions, and predicts next-step tokens using the LLM head. These tokens are decoded into ego trajectories, multi-agent trajectories, and language answers. The process repeats for $T$ steps, with updated scenario descriptions and ego-action questions at rollout each step, enabling consistent rollouts aligned with text and environment dynamics. Rollouts are optimized using the consistency loss $\mathcal{L}_{\text{roll}}$ and the reinforcement feedback loss $\mathcal{L}_{\text{GRPO}}$ for robust planning and reasoning.
• Figure 4: Qualitative examples of language-guided scenario rollouts using proposed GeRo on Bench2Drive. Top: Intersection scenarios with STOP signs and pedestrian interactions, where the ego vehicle demonstrates cautious deceleration, yielding, and safe left-turn. Bottom: Accident-handling scenarios under adverse weather, showing proactive hazard detection, lane-change planning, and smooth merging. Each frame includes generated textual reasoning aligned with the ego actions, highlighting temporal consistency and safety-aware decision.