FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder

TL;DR

FROST-Drive demonstrates that freezing a powerful Vision-Language Model vision encoder preserves broad world knowledge and high-dimensional representations, enabling robust end-to-end driving without full fine-tuning. The architecture fuses frozen-VLM features with a transformer adapter and a GRU decoder, optimized by a novel RFS loss that emphasizes safe trajectory planning in a speed-aware trust region. Empirical results on Waymo Open E2E show that frozen VLM encoders outperform fine-tuned baselines, with performance improving as VLM size and embedding dimensionality increase, achieving competitive leaderboard standings. This work suggests a scalable pathway for vision-based autonomy that leverages large foundation models for robustness in real-world, long-tail driving scenarios.

Abstract

End-to-end (E2E) models in autonomous driving aim to directly map sensor inputs to control commands, but their ability to generalize to novel and complex scenarios remains a key challenge. The common practice of fully fine-tuning the vision encoder on driving datasets potentially limits its generalization by causing the model to specialize too heavily in the training data. This work challenges the necessity of this training paradigm. We propose FROST-Drive, a novel E2E architecture designed to preserve and leverage the powerful generalization capabilities of a pretrained vision encoder from a Vision-Language Model (VLM). By keeping the encoder's weights frozen, our approach directly transfers the rich, generalized world knowledge from the VLM to the driving task. Our model architecture combines this frozen encoder with a transformer-based adapter for multimodal fusion and a GRU-based decoder for smooth waypoint generation. Furthermore, we introduce a custom loss function designed to directly optimize for Rater Feedback Score (RFS), a metric that prioritizes robust trajectory planning. We conduct extensive experiments on Waymo Open E2E Dataset, a large-scale datasets deliberately curated to capture the long-tail scenarios, demonstrating that our frozen-encoder approach significantly outperforms models that employ full fine-tuning. Our results provide substantial evidence that preserving the broad knowledge of a capable VLM is a more effective strategy for achieving robust, generalizable driving performance than intensive domain-specific adaptation. This offers a new pathway for developing vision-based models that can better handle the complexities of real-world application domains.

Figures (7)

• Figure 1: Comparison of the model performance for different vision encoder approaches on Waymo E2E Dataset; ViT (ImageNet): use a frozen ViT pre-trained with ImageNet dataset; ViT (Finetune): fine-tune the E2E model end-to-end with Waymo Dataset; VLM: use a frozen ViT from a 72B VLM.
• Figure 2: Comparison of the model performance using different sizes of visual embeddings on Waymo E2E Dataset; X axis: end-to-end model using a ViT with different embedding sizes.
• Figure 3: Model Architecture
• Figure 4: GRU-based Decoder Architecture.
• Figure 5: Example of raw inputs provided in the dataset.