Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving

Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, Liam Paull

TL;DR

The paper addresses robust end-to-end autonomous driving in long-tail and out-of-distribution scenarios by leveraging a simple, off-the-shelf 3B vision-language model trained with vision-language-trajectory (VLT) pre-training and lightweight reinforcement-learning post-training (GRPO). Using automatically generated language annotations, Poutine learns base driving behavior from CoVLA and WOD-E2E data and is refined with GRPO using a small set of human preferences, achieving state-of-the-art results on the Waymo WOD-E2E benchmark (7.99 RFS) and demonstrating zero-shot transfer from Japanese to US driving. The work shows that scalable VLT pre-training combined with minimal RL fine-tuning can yield robust, generalizable autonomous driving without bespoke perception backbones or trajectory heads. It highlights practical benefits for long-tail safety-critical scenarios and sets a new benchmark for VLM-based E2E driving.

Abstract

Maintaining good driving behavior in out-of-distribution scenarios remains a critical challenge in autonomous driving. A promising direction is to leverage the generalist knowledge and reasoning capabilities of large-language models by treating unusual driving scenarios as a logical reasoning task. In this work, we present Poutine, a method that uses an off-the-shelf 3B-parameter vision-language model (VLM) - without any additional components - to achieve robust end-to-end autonomous driving via a simple and scalable training recipe. To learn strong base driving capabilities, we first train Poutine-Base using self-supervised next-token prediction over vision, language, and trajectory (VLT) tokens, leveraging both nominal and long-tail driving data. In the second stage, we fine-tune Poutine-Base using Group Relative Policy Optimization (GRPO) with a small set of human preference-labeled examples. We evaluated our approach on the Waymo end-to-end driving benchmark curated for long-tail scenarios. The final Poutine model achieves an RFS of 7.99 on the test set, placing 1st in the 2025 Waymo Vision-Based End-to-End Driving Challenge by a significant margin. Our results suggest that handcrafted tokenizers or custom architectural components added to base VLMs in prior work are not necessary to achieve strong driving performance. Instead, this work highlights the potential of scalable VLT pretraining combined with lightweight RL fine-tuning to enable robust and generalizable autonomous driving.

Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving

TL;DR

The paper addresses robust end-to-end autonomous driving in long-tail and out-of-distribution scenarios by leveraging a simple, off-the-shelf 3B vision-language model trained with vision-language-trajectory (VLT) pre-training and lightweight reinforcement-learning post-training (GRPO). Using automatically generated language annotations, Poutine learns base driving behavior from CoVLA and WOD-E2E data and is refined with GRPO using a small set of human preferences, achieving state-of-the-art results on the Waymo WOD-E2E benchmark (7.99 RFS) and demonstrating zero-shot transfer from Japanese to US driving. The work shows that scalable VLT pre-training combined with minimal RL fine-tuning can yield robust, generalizable autonomous driving without bespoke perception backbones or trajectory heads. It highlights practical benefits for long-tail safety-critical scenarios and sets a new benchmark for VLM-based E2E driving.

Abstract

Maintaining good driving behavior in out-of-distribution scenarios remains a critical challenge in autonomous driving. A promising direction is to leverage the generalist knowledge and reasoning capabilities of large-language models by treating unusual driving scenarios as a logical reasoning task. In this work, we present Poutine, a method that uses an off-the-shelf 3B-parameter vision-language model (VLM) - without any additional components - to achieve robust end-to-end autonomous driving via a simple and scalable training recipe. To learn strong base driving capabilities, we first train Poutine-Base using self-supervised next-token prediction over vision, language, and trajectory (VLT) tokens, leveraging both nominal and long-tail driving data. In the second stage, we fine-tune Poutine-Base using Group Relative Policy Optimization (GRPO) with a small set of human preference-labeled examples. We evaluated our approach on the Waymo end-to-end driving benchmark curated for long-tail scenarios. The final Poutine model achieves an RFS of 7.99 on the test set, placing 1st in the 2025 Waymo Vision-Based End-to-End Driving Challenge by a significant margin. Our results suggest that handcrafted tokenizers or custom architectural components added to base VLMs in prior work are not necessary to achieve strong driving performance. Instead, this work highlights the potential of scalable VLT pretraining combined with lightweight RL fine-tuning to enable robust and generalizable autonomous driving.

Paper Structure

This paper contains 10 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Poutine
  4. Language Annotation Dataset Collection
  5. Vision-Language-Trajectory Pre-Training
  6. Reinforcement Learning Post-Training
  7. Results
  8. Conclusion
  9. Appendix
  10. System prompts

Figures (5)

  • Figure 1: (Top row) The Poutine model adopts an off-the-shelf 3B-parameter VLM with no additional components, and simply encodes trajectories in text space. (Bottom row) The proposed pipeline to train an end-to-end planner for robust autonomous driving. Datasets shown in the figure (i.e., CoVLA arai2025covla and WOD-E2E waymo2025e2e) indicate those used in our experiments. Fine-tuning of the Poutine model follows the highlighted stages with the vision encoder kept frozen.
  • Figure 2: Generated annotations on WOD-E2E data. The red dots depict the 5-second future trajectory. Only two views from the current frame and the driving description of the annotation are shown. Bold text highlights the objects and meta behavior selected by the model.
  • Figure 3: GRPO Results. Comparison between RL on a model pretrained with (blue) versus without (green) CoVLA. Both models were pre-trained on WOD-E2E training data. Checkpoints were evaluated on a held-out test set of 63 examples from the WOD-E2E validation split.
  • Figure 4: System prompt for generating language annotations. We changed 'driving scenarios' to 'left-hand-side driving scenarios' and 'multi-view images' to 'front-view images' for CoVLA.
  • Figure 5: System prompt used for VLT pre-training. (Top) Prompt used for frames with language annotations. (Bottom) Prompt used for frames without language annotations. This is also the prompt used during the RL post-training stage. The prompts were adjusted accordingly for pre-training on CoVLA data.