Unleashing the Capabilities of Large Vision-Language Models for Intelligent Perception of Roadside Infrastructure

TL;DR

This work tackles the gap between general-purpose vision-language models and the domain-specific, attribute-rich perception required for urban roadside infrastructure. It introduces a domain-adapted pipeline that combines open-vocabulary detection (Grounding DINO) with LoRA-finetuned Qwen-VL for attribute reasoning, augmented by a dual-modality Retrieval-Augmented Generation (RAG) to ground outputs in industry standards and visual exemplars. The approach yields 58.9 mAP on roadside detection and 95.5% attribute accuracy on Shanghai data, with strong cross-city generalization, demonstrating practical viability for intelligent infrastructure monitoring. Outputs are structured JSON and support interactive querying, enabling seamless integration into maintenance pipelines. Future work explores temporal reasoning in video to track asset states over time.

Abstract

Automated perception of urban roadside infrastructure is crucial for smart city management, yet general-purpose models often struggle to capture the necessary fine-grained attributes and domain rules. While Large Vision Language Models (VLMs) excel at open-world recognition, they often struggle to accurately interpret complex facility states in compliance with engineering standards, leading to unreliable performance in real-world applications. To address this, we propose a domain-adapted framework that transforms VLMs into specialized agents for intelligent infrastructure analysis. Our approach integrates a data-efficient fine-tuning strategy with a knowledge-grounded reasoning mechanism. Specifically, we leverage open-vocabulary fine-tuning on Grounding DINO to robustly localize diverse assets with minimal supervision, followed by LoRA-based adaptation on Qwen-VL for deep semantic attribute reasoning. To mitigate hallucinations and enforce professional compliance, we introduce a dual-modality Retrieval-Augmented Generation (RAG) module that dynamically retrieves authoritative industry standards and visual exemplars during inference. Evaluated on a comprehensive new dataset of urban roadside scenes, our framework achieves a detection performance of 58.9 mAP and an attribute recognition accuracy of 95.5%, demonstrating a robust solution for intelligent infrastructure monitoring.

Figures (12)

• Figure 1: Workflow of the proposed attribute- and state-aware open-vocabulary vision-language framework.
• Figure 2: Example of structured JSON representation for roadside object attributes.
• Figure 3: Model architectures of the three fine-tuning strategies.
• Figure 4: Overview of the dual Retrieval-Augmented Generation (RAG) framework, integrating Textual RAG for knowledge-grounded reasoning using domain standards, and Visual RAG for exemplar-based attribute retrieval from annotated image databases.
• Figure 5: A case study of the RAG-enhanced inference process. The system retrieves the "Mandatory sign" definition from GB 5768.2–2022 and visually similar exemplars to guide the generation of structured attribute outputs for a blue circular traffic sign.