STER-VLM: Spatio-Temporal With Enhanced Reference Vision-Language Models

TL;DR

The paper tackles the resource-heavy nature of vision-language models in traffic analysis by introducing STER-VLM, a framework that achieves strong spatio-temporal understanding with efficiency. It achieves this through caption decomposition into spatial and temporal components, prudent frame selection with best-view filtering, reference-driven guidance, and carefully crafted visual/textual prompts, all built on LoRA-tuned Qwen models. Empirical results on WTS and BDD show robust captioning and VQA performance, with an AI City Challenge score of 55.655 and a top-10 leaderboard standing, underscoring practical impact for real-world traffic safety. The work demonstrates that decomposed representations and guided prompting can yield high-quality, context-aware traffic analysis while staying computationally economical.

Abstract

Vision-language models (VLMs) have emerged as powerful tools for enabling automated traffic analysis; however, current approaches often demand substantial computational resources and struggle with fine-grained spatio-temporal understanding. This paper introduces STER-VLM, a computationally efficient framework that enhances VLM performance through (1) caption decomposition to tackle spatial and temporal information separately, (2) temporal frame selection with best-view filtering for sufficient temporal information, and (3) reference-driven understanding for capturing fine-grained motion and dynamic context and (4) curated visual/textual prompt techniques. Experimental results on the WTS \cite{kong2024wts} and BDD \cite{BDD} datasets demonstrate substantial gains in semantic richness and traffic scene interpretation. Our framework is validated through a decent test score of 55.655 in the AI City Challenge 2025 Track 2, showing its effectiveness in advancing resource-efficient and accurate traffic analysis for real-world applications.

Figures (5)

• Figure 1: The overview of the our pipeline. We employ Qwen2.5-7B-Instruct bai2025qwen25vltechnicalreport, enhanced with LoRA hu2022lora, and incorporate a combination of frame selection and filtering, textual-visual prompting and a caption decomposition training strategy.
• Figure 2: An illustration of our caption decomposition strategy. Red highlights spatial-invariant details (e.g., environment, object attributes), while blue indicates temporal-variant cues (e.g., actions, positions).
• Figure 3: Illustration of our frame selection strategy. The first two frames capture the pedestrian's appearance, while the last two provide environmental context. Using only a single first frame for inference leads to inaccurate descriptions due to limited visual evidence. In contrast, incorporating four frames results in more accurate and coherent captions. Temporal-variant components are excluded here for clearer visualization.
• Figure 4: Examples of visual prompts (Left) and corresponding textual prompts (Right).
• Figure 5: Examples of captions with correct references (Left) and misinterpreted references (Right) from LVLM.