TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning

TL;DR

TrafficVLM tackles dense, multi-target captioning of traffic videos by combining sub-global and local visual features with a temporal encoder and a T5-based decoder to produce long, phase-aware captions for vehicles and pedestrians. The model reframes traffic safety analysis as joint temporal localization and dense captioning, with a controllable generation component and a two-task fine-tuning objective. Evaluated on the AI City Challenge Track 2 WTS dataset, it achieves third place, demonstrating strong performance across vehicle ego and overhead views. The approach advances practical traffic video understanding and offers a path toward traffic QA, summarization, and more controllable video-language systems.

Abstract

Traffic video description and analysis have received much attention recently due to the growing demand for efficient and reliable urban surveillance systems. Most existing methods only focus on locating traffic event segments, which severely lack descriptive details related to the behaviour and context of all the subjects of interest in the events. In this paper, we present TrafficVLM, a novel multi-modal dense video captioning model for vehicle ego camera view. TrafficVLM models traffic video events at different levels of analysis, both spatially and temporally, and generates long fine-grained descriptions for the vehicle and pedestrian at different phases of the event. We also propose a conditional component for TrafficVLM to control the generation outputs and a multi-task fine-tuning paradigm to enhance TrafficVLM's learning capability. Experiments show that TrafficVLM performs well on both vehicle and overhead camera views. Our solution achieved outstanding results in Track 2 of the AI City Challenge 2024, ranking us third in the challenge standings. Our code is publicly available at https://github.com/quangminhdinh/TrafficVLM.

Figures (2)

• Figure 1: Overview of our method. First the sub-global and local frame sequences are extracted from the vehicle camera video. Some local frames might be missing, depending on the availability of the bounding boxes in the event segment. A visual feature extractor $f_e$ is then applied to both of them to get the two visual embeddings $x^g$ and parts of $x^l$. The sub-global embedding is trimmed to the event segment and subsampled to create the sub-global feature $\tilde{x^g}$. Feature vectors in the learnable local tensor $u$ are added to the local embedding as the embeddings for the missing phases. Positional embeddings are then applied to both visual features, followed by the temporal encoder $f_t$. The final embeddings $z^g$ and $z^l$ are concatenated with the conditional embedding $z^c_v$ for vehicle or $z^c_p$ for pedestrian to control the generation output. The text decoder $h$ receives the concatenated embedding as the input and autoregressively generates the output sequence. For fine-tuning, the final loss is calculated by combining the losses for generating the vehicle and pedestrian output sequences.
• Figure 2: Sub-global and local segments for vehicle and overhead camera views.