Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Models

Yiru Wang, Anqing Jiang, Shuo Wang, Yuwen Heng, Zichong Gu, Hao Sun

Abstract

The integration of Vision-Language-Action (VLA) models into autonomous driving systems offers a unified framework for interpreting complex scenes and executing control commands. However, the necessity to incorporate historical multi-view frames for accurate temporal reasoning imposes a severe computational burden, primarily driven by the quadratic complexity of self-attention mechanisms in Large Language Models (LLMs). To alleviate this bottleneck, we propose ETA-VLA, an Efficient Token Adaptation framework for VLA models. ETA-VLA processes the past $n$ frames of multi-view images and introduces a novel Intra-LLM Sparse Aggregator (ILSA). Drawing inspiration from human driver attention allocation, ILSA dynamically identifies and prunes redundant visual tokens guided by textual queries and temporal consistency. Specifically, we utilize a text-guided scoring mechanism alongside a diversity-preserving sparsification strategy to select a sparse subset of critical tokens, ensuring comprehensive awareness of the driving scene. Extensive experiments on the NAVSIM v2 demonstrate that ETA-VLA achieves driving performance comparable to state-of-the-art baselines while reducing computational FLOPs by approximately 32\%. Notably, our method prunes 85% of visual tokens and reduces inference FLOPs by 61\%, but still retaining 94% of the original accuracy on the NAVSIM v2 benchmark.

ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Models

Abstract

The integration of Vision-Language-Action (VLA) models into autonomous driving systems offers a unified framework for interpreting complex scenes and executing control commands. However, the necessity to incorporate historical multi-view frames for accurate temporal reasoning imposes a severe computational burden, primarily driven by the quadratic complexity of self-attention mechanisms in Large Language Models (LLMs). To alleviate this bottleneck, we propose ETA-VLA, an Efficient Token Adaptation framework for VLA models. ETA-VLA processes the past frames of multi-view images and introduces a novel Intra-LLM Sparse Aggregator (ILSA). Drawing inspiration from human driver attention allocation, ILSA dynamically identifies and prunes redundant visual tokens guided by textual queries and temporal consistency. Specifically, we utilize a text-guided scoring mechanism alongside a diversity-preserving sparsification strategy to select a sparse subset of critical tokens, ensuring comprehensive awareness of the driving scene. Extensive experiments on the NAVSIM v2 demonstrate that ETA-VLA achieves driving performance comparable to state-of-the-art baselines while reducing computational FLOPs by approximately 32\%. Notably, our method prunes 85% of visual tokens and reduces inference FLOPs by 61\%, but still retaining 94% of the original accuracy on the NAVSIM v2 benchmark.

Paper Structure

This paper contains 28 sections, 9 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Vision-Language-Action Models
  4. Visual Token Sparsification in VLMs
  5. Temporal Modeling in Autonomous Driving
  6. METHODOLOGY
  7. System Overview
  8. Temporal Fusion Module (TFM)
  9. Intra-LLM Sparse Aggregator (ILSA)
  10. ILSA Overview
  11. Hybrid Execution Strategy
  12. Dynamic Text Anchor Selection
  13. RoPE-free Semantic Scoring
  14. Diversity-Preserving Token Selection
  15. Global Semantic-Relative Pruning:
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Visualization of ILSA's token pruning and recycling: The left column shows the front-view camera input with the predicted trajectory (green line) and safety-critical objects (orange boxes). The right columns display multi-view images (ordered as Front, Front Left, Rear Left, Front Right, Rear Right, and Rear) with pruning masks (black boxes), recycled tokens (green boxes), and safety-critical objects (orange boxes). Our method (top) prioritizes critical regions and maintains diversity, mimicking human attention allocation. In contrast, the per-view average method (bottom) dilutes focus.
  • Figure 2: ETA-VLA Framework with Temporal Fusion Module (TFM), Intra-LLM Sparse Aggregator (ILSA) and training stage.
  • Figure 3: The Temporal Fusion Module (TFM) for adaptive historical multi-view information aggregation.
  • Figure 4: Intra-LLM Sparse Aggregator (ILSA) Framework with Semantic-Relative Pruning and Diversity-Preserving Recycling.