TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning

TL;DR

TS-VLM tackles the inefficiency of existing vision-language models for multi-view driving by introducing TGSSP, a text-guided SoftSort pooling module that performs query-aware fusion across camera views without heavy attention. It employs a MobileViTv2-based lightweight image encoder and T5 variants as the language model, achieving strong driving-reasoning performance on DriveLM-nuScenes with only 20.1M parameters and about 0.11 GB memory during inference. On DriveLM-nuScenes, TS-VLM delivers BLEU-4 56.82, METEOR 41.91, ROUGE-L 74.64, and CIDEr 3.39, while reducing computation up to 90% compared with larger models. This work demonstrates that differentiable, query-guided multi-view fusion can yield high semantic understanding with real-time feasibility, enabling edge deployment of vision-language agents for autonomous driving.

Abstract

Vision-Language Models (VLMs) have shown remarkable potential in advancing autonomous driving by leveraging multi-modal fusion in order to enhance scene perception, reasoning, and decision-making. Despite their potential, existing models suffer from computational overhead and inefficient integration of multi-view sensor data that make them impractical for real-time deployment in safety-critical autonomous driving applications. To address these shortcomings, this paper is devoted to designing a lightweight VLM called TS-VLM, which incorporates a novel Text-Guided SoftSort Pooling (TGSSP) module. By resorting to semantics of the input queries, TGSSP ranks and fuses visual features from multiple views, enabling dynamic and query-aware multi-view aggregation without reliance on costly attention mechanisms. This design ensures the query-adaptive prioritization of semantically related views, which leads to improved contextual accuracy in multi-view reasoning for autonomous driving. Extensive evaluations on the DriveLM benchmark demonstrate that, on the one hand, TS-VLM outperforms state-of-the-art models with a BLEU-4 score of 56.82, METEOR of 41.91, ROUGE-L of 74.64, and CIDEr of 3.39. On the other hand, TS-VLM reduces computational cost by up to 90%, where the smallest version contains only 20.1 million parameters, making it more practical for real-time deployment in autonomous vehicles.

Figures (6)

• Figure 1: Model performance vs. model size on the DriveLM benchmark across four metrics: BLEU-4, METEOR, ROUGE-L, and CIDEr. Each circle represents a model, where the x-axis indicates model size (in millions of parameters, log scale), and the y-axis shows the performance score (↑ = better).
• Figure 2: Examples of perception, prediction, and planning tasks in the DriveLM-nuScenes dataset.
• Figure 3: The overall architecture of TS-VLM. Multi-view images are first encoded via the image encoder and then adaptively fused with textual features using the Text-Guided SoftSort Pooling (TGSSP) module. The resulting multimodal embeddings are fed into the $T5$ LLM to generate semantically accurate textual answers based on the given question.
• Figure 4: Structure of the proposed image encoder for TS-VLM.
• Figure 5: TGSSP Module.