Efficient Visual Question Answering Pipeline for Autonomous Driving via Scene Region Compression

TL;DR

This work tackles the real-time requirements of autonomous-driving Visual Question Answering by introducing SRC-Pipeline, a token compression strategy that replaces early-frame patch tokens with scene-region compression tokens while retaining dense late-frame patches. A novel SRC-ViT outputs one scene token and four region tokens per frame, enabling significant FLOPs reduction (around 66%) with only modest accuracy trade-offs. A two-stage training regime—Stage One for SRC-ViT tokenization using scene/region captions and Stage Two for adapting the VLM decoder to compressed tokens—enables seamless multimodal reasoning with limited computation. Experiments on LingoQA demonstrate the approach’s effectiveness and the importance of late-frame detail, underscoring its potential for real-time, safety-critical autonomous driving applications.

Abstract

Autonomous driving increasingly relies on Visual Question Answering (VQA) to enable vehicles to understand complex surroundings by analyzing visual inputs and textual queries. Currently, a paramount concern for VQA in this domain is the stringent requirement for fast latency and real-time processing, as delays directly impact real-world safety in this safety-critical application. However, current state-of-the-art VQA models, particularly large vision-language models (VLMs), often prioritize performance over computational efficiency. These models typically process dense patch tokens for every frame, leading to prohibitive computational costs (FLOPs) and significant inference latency, especially with long video sequences. This focus limits their practical deployment in real-time autonomous driving scenarios. To tackle this issue, we propose an efficient VLM framework for autonomous driving VQA tasks, SRC-Pipeline. It learns to compress early frame tokens into a small number of high-level tokens while retaining full patch tokens for recent frames. Experiments on autonomous driving video question answering tasks show that our approach achieves 66% FLOPs reduction while maintaining comparable performance, enabling VLMs to operate more effectively in real-time, safety-critical autonomous driving settings.

Figures (3)

• Figure 1: Difference between general-purpose video question answering and autonomous driving video question answering. While the question of the general-purpose video question answering covers the information from all the frames, the question of the autonomous driving video question answering mostly focuses on the later frames due to the timeliness of the autonomous driving video.
• Figure 2: The architecture of SRC-Pipeline. (left): The overall architecture of SRC-Pipeline. Video frames are sent to the SRC-ViT to produce both the scene-region token and the patch token, where the early frames are represented by the scene-region tokens and the later frames are represented by the patch tokens. (upper right): The architecture of the SRC-ViT. Learnable scene and region tokens are attached to the path tokens from linear projection, and the positional embedding is applied to both the scene region tokens and the patch tokens. (lower left): The region attention mask. For each of the region tokens, the attention of it to the patch tokens that do not belong to the specific region will be masked out.
• Figure 3: The stage one training pipeline. The SRC-ViT is paired with CLIP's text encoder. For every image, one scene caption and four region captions are encoded via the text encoder, and the [CLS] token of each is aligned with the image's scene token and region tokens for contrastive learning.