Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning

Jiajun Cao, Qizhe Zhang, Peidong Jia, Xuhui Zhao, Bo Lan, Xiaoan Zhang, Zhuo Li, Xiaobao Wei, Sixiang Chen, Liyun Li, Xianming Liu, Ming Lu, Yang Wang, Shanghang Zhang

TL;DR

This work tackles the high computational cost of Vision-Language-Action (VLA) models in end-to-end autonomous driving by introducing FastDriveVLA, a reconstruction-based token pruning framework. A plug-and-play ReconPruner prioritizes foreground information through MAE-style pixel reconstruction and is trained with an adversarial foreground-background reconstruction strategy, enabling robust token selection without retraining for new models. The authors also release nuScenes-FG, a large-scale foreground-annotated dataset, to support foreground-aware training. Empirical results on nuScenes open-loop planning show state-of-the-art performance across pruning ratios, along with significant efficiency gains, demonstrating the practicality and effectiveness of foreground-focused pruning for driving tasks.

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential in complex scene understanding and action reasoning, leading to their increasing adoption in end-to-end autonomous driving systems. However, the long visual tokens of VLA models greatly increase computational costs. Current visual token pruning methods in Vision-Language Models (VLM) rely on either visual token similarity or visual-text attention, but both have shown poor performance in autonomous driving scenarios. Given that human drivers concentrate on relevant foreground areas while driving, we assert that retaining visual tokens containing this foreground information is essential for effective decision-making. Inspired by this, we propose FastDriveVLA, a novel reconstruction-based vision token pruning framework designed specifically for autonomous driving. FastDriveVLA includes a plug-and-play visual token pruner called ReconPruner, which prioritizes foreground information through MAE-style pixel reconstruction. A novel adversarial foreground-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models. Once trained, ReconPruner can be seamlessly applied to different VLA models with the same visual encoder without retraining. To train ReconPruner, we also introduce a large-scale dataset called nuScenes-FG, consisting of 241K image-mask pairs with annotated foreground regions. Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.

FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning

TL;DR

This work tackles the high computational cost of Vision-Language-Action (VLA) models in end-to-end autonomous driving by introducing FastDriveVLA, a reconstruction-based token pruning framework. A plug-and-play ReconPruner prioritizes foreground information through MAE-style pixel reconstruction and is trained with an adversarial foreground-background reconstruction strategy, enabling robust token selection without retraining for new models. The authors also release nuScenes-FG, a large-scale foreground-annotated dataset, to support foreground-aware training. Empirical results on nuScenes open-loop planning show state-of-the-art performance across pruning ratios, along with significant efficiency gains, demonstrating the practicality and effectiveness of foreground-focused pruning for driving tasks.

Abstract

Vision-Language-Action (VLA) models have demonstrated significant potential in complex scene understanding and action reasoning, leading to their increasing adoption in end-to-end autonomous driving systems. However, the long visual tokens of VLA models greatly increase computational costs. Current visual token pruning methods in Vision-Language Models (VLM) rely on either visual token similarity or visual-text attention, but both have shown poor performance in autonomous driving scenarios. Given that human drivers concentrate on relevant foreground areas while driving, we assert that retaining visual tokens containing this foreground information is essential for effective decision-making. Inspired by this, we propose FastDriveVLA, a novel reconstruction-based vision token pruning framework designed specifically for autonomous driving. FastDriveVLA includes a plug-and-play visual token pruner called ReconPruner, which prioritizes foreground information through MAE-style pixel reconstruction. A novel adversarial foreground-background reconstruction strategy is designed to train ReconPruner for the visual encoder of VLA models. Once trained, ReconPruner can be seamlessly applied to different VLA models with the same visual encoder without retraining. To train ReconPruner, we also introduce a large-scale dataset called nuScenes-FG, consisting of 241K image-mask pairs with annotated foreground regions. Our approach achieves state-of-the-art results on the nuScenes open-loop planning benchmark across different pruning ratios.

Paper Structure

This paper contains 21 sections, 10 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. nuScenes-FG Dataset
  5. ReconPruner: Reconstruction-based Pruner
  6. Adversarial Foreground-Background Reconstruction Strategy
  7. Training Loss
  8. Pruning During Inference
  9. Experiments
  10. Experimental Settings
  11. Models.
  12. Datasets and Metrics.
  13. Baselines.
  14. Training.
  15. Evaluation on the nuScenes
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Comparison of different visual token pruning strategies.
  • Figure 2: nuScenes-FG. It contains 241k foreground segmentation annotations for scenes in the nuScenes dataset.
  • Figure 3: FastDriveVLA framework. During training, a novel adversarial foreground-background reconstruction strategy is proposed to enhance ReconPruner’s ability to perceive foreground visual tokens. During inference, ReconPruner can be directly integrated into autonomous driving VLA models for token pruning.
  • Figure 4: Visualization of reconstruction.
  • Figure 5: Visual comparison of visual tokens retained by different visual token pruning methods.