Temporal Object Captioning for Street Scene Videos from LiDAR Tracks

TL;DR

This work tackles the lack of temporally grounded descriptions in street-scene captions by introducing a fully automated LiDAR-based captioning pipeline. It converts LiDAR object tracks into host and neighbor captions through a rule-based, template-driven system, producing temporally rich descriptions that are used to supervise a video captioning model trained on front-camera frames. The SwinBERT model, trained with these captions and augmented with object masks, demonstrates improved temporal understanding and reduced visual bias, quantified by the Visual Bias Measure $VBM = 100 \cdot \frac{B_k - C_k}{B_k}$, across proprietary data as well as public datasets Waymo and NuScenes. The approach yields strong zero-shot generalization, clearer embedding structure (as shown by UMAP), and scalable data generation without human annotations, making it practical for real-world autonomous driving systems.

Abstract

Video captioning models have seen notable advancements in recent years, especially with regard to their ability to capture temporal information. While many research efforts have focused on architectural advancements, such as temporal attention mechanisms, there remains a notable gap in understanding how models capture and utilize temporal semantics for effective temporal feature extraction, especially in the context of Advanced Driver Assistance Systems. We propose an automated LiDAR-based captioning procedure that focuses on the temporal dynamics of traffic participants. Our approach uses a rule-based system to extract essential details such as lane position and relative motion from object tracks, followed by a template-based caption generation. Our findings show that training SwinBERT, a video captioning model, using only front camera images and supervised with our template-based captions, specifically designed to encapsulate fine-grained temporal behavior, leads to improved temporal understanding consistently across three datasets. In conclusion, our results clearly demonstrate that integrating LiDAR-based caption supervision significantly enhances temporal understanding, effectively addressing and reducing the inherent visual/static biases prevalent in current state-of-the-art model architectures.

Figures (8)

• Figure 1: LiDAR-based caption generation and video captioning model training. This figure shows the overall workflow of our approach. Object tracks extracted from raw LiDAR data using a SOTA 3D detector and tracker are used to identify event boundaries and generate object-level captions. The object caption and associated front camera RGB images (using event boundaries) are used to train a video captioning model (multi-modal masked language modeling).
• Figure 2: LiDAR-based caption generation and video captioning model training. This figure illustrates our overall caption generation procedure and the training process for the video captioning model. Using the host vehicle's sensor data (speed and yaw rate) and object tracks from LiDAR data, we generate the object, lane and motion tags. These tags are used in the corresponding placeholders of the template sentence to construct the host and object captions. Finally, the object captions with the front camera RGB images are used to train a video captioning model (SwinBERT)through Masked Language Modeling. VT, WT, and MMT stand for Vision transformer, Word tokenizer, and Multi-modal transformer, respectively.
• Figure 3: LiDAR-based object mask generation. This figure highlights our mask generation procedure. For a given input frame, the point clouds corresponding to the object of interest (black car) are first projected onto the front-camera image plane, forming a dense 2-D scatter (center image in the green box). Andrew’s Monotone Chain convex-hull algorithm then traces a tight contour around these projected points, which is rasterised into a mask that cleanly outlines the object (right image in the green box).
• Figure 4: Thresholds for lane tag. This figure illustrates the threshold values used in our neighbor captioning process. On the left, we depict the yaw angle thresholds applied to bounding boxes for lane tagging. On the right, we show the thresholds used for determining lane tags based on bounding box center positions.
• Figure 5: Visual Bias Measure (VBM) based comparison. A comparison of our model, trained using our proprietary dataset, with several SOTA models via a video-to-video retrieval approach, evaluated with our custom VBM metric across three datasets. Left and right plots illustrate the benchmarking results using mask-augmented frames and raw frames, respectively. A lower VBM score indicates that the model's embeddings are less influenced by visual bias present in the video.