SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understanding

TL;DR

SafePLUG advances traffic accident understanding by equipping multimodal LLMs with pixel-level reasoning and temporal grounding. It introduces a dual-LoRA architecture and a visual-number prompt system to enable region-aware QA, pixel-level segmentation, and temporally anchored event localization, all fused within a single LLM backbone and SAM-based decoder. The SafePLUG-Bench dataset provides over 220K multimodal QA pairs with region and pixel-level annotations, facilitating comprehensive evaluation. Across region QA, pixel grounding, accident description, and temporal localization, SafePLUG outperforms strong baselines and ablations confirm the value of modular, task-specific adapters and input prompts. The work enables fine-grained, temporally coherent understanding of complex traffic scenes, with potential impacts on safety analysis, anomaly detection, and smart transportation systems.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress across a range of vision-language tasks and demonstrate strong potential for traffic accident understanding. However, existing MLLMs in this domain primarily focus on coarse-grained image-level or video-level comprehension and often struggle to handle fine-grained visual details or localized scene components, limiting their applicability in complex accident scenarios. To address these limitations, we propose SafePLUG, a novel framework that empowers MLLMs with both Pixel-Level Understanding and temporal Grounding for comprehensive traffic accident analysis. SafePLUG supports both arbitrary-shaped visual prompts for region-aware question answering and pixel-level segmentation based on language instructions, while also enabling the recognition of temporally anchored events in traffic accident scenarios. To advance the development of MLLMs for traffic accident understanding, we curate a new dataset containing multimodal question-answer pairs centered on diverse accident scenarios, with detailed pixel-level annotations and temporal event boundaries. Experimental results show that SafePLUG achieves strong performance on multiple tasks, including region-based question answering, pixel-level segmentation, temporal event localization, and accident event understanding. These capabilities lay a foundation for fine-grained understanding of complex traffic scenes, with the potential to improve driving safety and enhance situational awareness in smart transportation systems. The code, dataset, and model checkpoints will be made publicly available at: https://zihaosheng.github.io/SafePLUG

Figures (9)

• Figure 1: SafePLUG supports both image/video-level and pixel-level understanding through accident description, temporal localization, region-level question answering, and pixel-level grounding, enabling comprehensive traffic accident analysis.
• Figure 2: Overview of SafePLUG. The model takes as input multiple modalities, including video frames with number prompts, image-level context, and user-defined visual prompts, and unifies them with language prompts through an LLM backbone. The features are then decoded into either natural language answers or binary segmentation masks.
• Figure 3: Semi-automated data annotation pipeline leveraging MLLMs and SAM for generating region-level descriptions, accident narratives, and segmentation masks.
• Figure 4: Data statistics of SafePLUG-Bench, including text length, video duration, accident time ratio, spatial region density, object distribution, and accident type distribution.
• Figure 5: Qualitative comparison of accident descriptions generated by different models. The highlighted colors indicate key causal phrases, agent actions, and event interpretations.