MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios

TL;DR

The paper tackles joint semantic scene understanding and risk localization in traffic using an image-based multimodal large language model. It introduces MLLM-SUL, featuring a dual-branch visual encoder (low-resolution grid features from ResNet-101 and high-resolution region features from a detector and Swin-L backbone) fused via a gate-attention mechanism, a LLaMA-2-7B-based caption reasoning module, and a regression module with a learnable query to predict risk-object coordinates. The model is trained in two stages and evaluated on DRAMA-ROLISP and the extended DRAMA-SRIS datasets, achieving state-of-the-art results in both captioning (BLEU/METEOR/CIDEr) and localization (mIoU, IoU>0.5) while maintaining efficiency compared to video-based baselines. A key contribution is the use of a dual-resolution visual encoder to provide rich, multi-scale information to the language model, paired with a regression head to avoid numeric errors in LLM outputs, enabling accurate risk description and precise localization in traffic scenarios, with code and data available publicly.

Abstract

Multimodal large language models (MLLMs) have shown satisfactory effects in many autonomous driving tasks. In this paper, MLLMs are utilized to solve joint semantic scene understanding and risk localization tasks, while only relying on front-view images. In the proposed MLLM-SUL framework, a dual-branch visual encoder is first designed to extract features from two resolutions, and rich visual information is conducive to the language model describing risk objects of different sizes accurately. Then for the language generation, LLaMA model is fine-tuned to predict scene descriptions, containing the type of driving scenario, actions of risk objects, and driving intentions and suggestions of ego-vehicle. Ultimately, a transformer-based network incorporating a regression token is trained to locate the risk objects. Extensive experiments on the existing DRAMA-ROLISP dataset and the extended DRAMA-SRIS dataset demonstrate that our method is efficient, surpassing many state-of-the-art image-based and video-based methods. Specifically, our method achieves 80.1% BLEU-1 score and 298.5% CIDEr score in the scene understanding task, and 59.6% accuracy in the localization task. Codes and datasets are available at https://github.com/fjq-tongji/MLLM-SUL.

Figures (5)

• Figure 1: An example of errors in MLLM. Wrong semantic descriptions and numbers are highlighted in red and blue, respectively.
• Figure 2: Overall structure of the proposed MLLM-SUL model. The prompts are: What is the current driving scenario? Which object is at the highest risk? Then predict the intentions and suggestions for the ego-car.
• Figure 3: The structure of the transformer-based regression module. (Purple blocks represent text sequences, and blue blocks represent image sequences.)
• Figure 4: Comparison of text descriptions for three traffic scenario dataset. The coordinate boxes for risk object detection in three datasets are the same.
• Figure 5: Qualitative results in the DRAMA-SRIS dataset. The language generator in BLIP-2 and InstructBLIP is OPT$_{\mathrm{6.7B}}$. "GT" indicates the ground-truth results and wrong predictions are represented in italics. For the same MLLM, descriptions and bounding box are represented by the same color.