Vision-Based Natural Language Scene Understanding for Autonomous Driving: An Extended Dataset and a New Model for Traffic Scene Description Generation

TL;DR

The paper tackles traffic scene understanding from a single image by casting image-to-text generation as $P(\boldsymbol{\mathbf{y}}|\boldsymbol{\mathbf{I}})$ and proposing a hybrid attention-based encoder–decoder framework augmented with a memory-enhanced decoder. It introduces a dedicated dataset derived from BDD100K, annotating 600 images with ten domain-focused sentences each, and evaluates with CIDEr and SPICE among other metrics to gauge semantic richness and descriptive accuracy. Key contributions include a formal theoretical framework, feature-preservation and memory-enhancement mechanisms, a multimodal fusion strategy, and an extensive ablation study that identifies MViTv2-S as an effective encoder and xLSTM as a capable decoder for generating detailed driving-related captions. The work demonstrates strong performance on the new dataset and lays groundwork for scaling to larger datasets, RL-based fine-tuning, and extending the approach to video-based traffic scene understanding with richer spatiotemporal descriptions.

Abstract

Traffic scene understanding is essential for enabling autonomous vehicles to accurately perceive and interpret their environment, thereby ensuring safe navigation. This paper presents a novel framework that transforms a single frontal-view camera image into a concise natural language description, effectively capturing spatial layouts, semantic relationships, and driving-relevant cues. The proposed model leverages a hybrid attention mechanism to enhance spatial and semantic feature extraction and integrates these features to generate contextually rich and detailed scene descriptions. To address the limited availability of specialized datasets in this domain, a new dataset derived from the BDD100K dataset has been developed, with comprehensive guidelines provided for its construction. Furthermore, the study offers an in-depth discussion of relevant evaluation metrics, identifying the most appropriate measures for this task. Extensive quantitative evaluations using metrics such as CIDEr and SPICE, complemented by human judgment assessments, demonstrate that the proposed model achieves strong performance and effectively fulfills its intended objectives on the newly developed dataset.

Figures (1)

• Figure 1: Overall description generation architecture for implementing the proposed framework.