LaVida Drive: Vision-Text Interaction VLM for Autonomous Driving with Token Selection, Recovery and Enhancement

TL;DR

LaVida Drive addresses the challenge of maintaining fine-grained, high-resolution perception in dynamic autonomous driving while managing computational costs. It introduces a two-stage approach: Query-aware Token Selection to filter image tokens by query relevance, and Spatial-temporal Token Enhancement to coherently fuse spatial and temporal information without adding tokens. The architecture leverages frozen CLIP encoders for images and text, TimeSformer for video, and a large language model for reasoning, reporting substantial token reductions (50–84%) and improved or competitive task performance on DriveLM and NuscenesQA benchmarks. Ablation studies confirm the importance of balancing token selection with compression and the value of multi-view, multi-frame inputs for robust, real-time VQA. Overall, LaVida Drive presents a scalable path toward efficient, high-fidelity vision-language reasoning in autonomous driving systems.

Abstract

Recent advancements in Visual Language Models (VLMs) have made them crucial for visual question answering (VQA) in autonomous driving, enabling natural human-vehicle interactions. However, existing methods often struggle in dynamic driving environments, as they usually focus on static images or videos and rely on downsampling to manage computational costs. This results in the loss of critical details and the difficulty in effectively integrating spatial and temporal information, undermining fine-grained perception and temporal coherence essential for effective decision-making. To tackle these challenges, we introduce LaVida Drive, a novel and efficient VQA framework for autonomous driving. LaVida Drive seamlessly integrates temporal data while maintaining high-resolution inputs for detailed visual perception. It optimizes spatial processing by retaining high-resolution data for intricate details and using lower-resolution inputs for temporal analysis to focus on motion-related features, thereby boosting computational efficiency. The core of LaVida Drive consists of two modules: the \textit{Query-aware Token Selection} module and the \textit{Spatial-Temporal Token Recovery and Enhancement} module. The former dynamically selects the most relevant visual tokens based on semantic alignment with the input query, reducing the token count from high-resolution spatial input. The latter ensures smooth and coherent interactions between spatial and temporal information, preserving contextual continuity across frames. Extensive experiments on various autonomous driving question-answering benchmarks show that LaVida Drive significantly reduces visual tokens, enhances efficiency, and improves overall performance.

Figures (4)

• Figure 1: Comparison between LaVida Drive and existing VLM-based autonomous driving methods. (Left): Traditional methods use a VIT projector with simple monocular input and downsampling. (Right): LaVida Drive efficiently captures high-resolution details while maintaining motion perception, replacing the projector with enhanced modules that leverage encoder-level tokens for spatial and temporal enhancement.
• Figure 2: Framework of LaVida Drive. High-resolution images are divided into $224\times 224$ patches and processed by the image encoder to extract semantic features. A token-level similarity matrix aligns image tokens with text tokens, enabling query-aware token selection. Selected tokens are enhanced through a token-wise attention mechanism in the enhancement module, utilizing auxiliary branches for restoration without increasing token count. The final tokens and original text embeddings are then fed into the large language model.
• Figure 3: The architecture of the Token-wise Attention module. We leverage the outputs from the support branches of the image encoder and video encoder, applying cross-attention to recover/enhance the selected tokens while maintaining consistent output dimensions according to the rules of attention computation.
• Figure 4: Lavida Drive Response Example. The areas not covered by the black mask represent the image patches corresponding to the retained tokens. This is provided as a simplified example for ease of visualization, and the actual results may differ.