SVII-3D: Advancing Roadside Infrastructure Inventory with Decimeter-level 3D Localization and Comprehension from Sparse Street Imagery

TL;DR

SVII-3D tackles the challenge of building high-fidelity 3D digital twins of roadside infrastructure from sparse, low-cost street imagery. It fuses LoRA-finetuned open-set detection with a spatial-attention matching network to robustly identify and cross-view associate assets, then applies a geometry-guided 3D localization pipeline for decimeter-level positioning. A state-discriminative Vision-Language Model (VLM) agent, enhanced with multimodal prompting, expert knowledge injection, and retrieval-augmented generation, provides fine-grained operational state descriptions to enrich asset inventories. Across Wuhan and Shanghai datasets, SVII-3D demonstrates strong 2D detection, robust cross-view matching, and precise 3D geo-localization, while maintaining cross-city generalizability and enabling automated maintenance insights through semantic state analysis.

Abstract

The automated creation of digital twins and precise asset inventories is a critical task in smart city construction and facility lifecycle management. However, utilizing cost-effective sparse imagery remains challenging due to limited robustness, inaccurate localization, and a lack of fine-grained state understanding. To address these limitations, SVII-3D, a unified framework for holistic asset digitization, is proposed. First, LoRA fine-tuned open-set detection is fused with a spatial-attention matching network to robustly associate observations across sparse views. Second, a geometry-guided refinement mechanism is introduced to resolve structural errors, achieving precise decimeter-level 3D localization. Third, transcending static geometric mapping, a Vision-Language Model agent leveraging multi-modal prompting is incorporated to automatically diagnose fine-grained operational states. Experiments demonstrate that SVII-3D significantly improves identification accuracy and minimizes localization errors. Consequently, this framework offers a scalable, cost-effective solution for high-fidelity infrastructure digitization, effectively bridging the gap between sparse perception and automated intelligent maintenance.

Figures (8)

• Figure 1: Overview of the proposed SVII-3D framework. The system operates in three stages: (I) Infrastructure Identification, combining LoRA fine-tuned detection and spatial-attention matching to associate sparse observations; (II) Geometry-guided 3D Localization, which refines associations for decimeter-level positioning; and (III) State-discriminative VLM Agent, inferring semantic attributes and operational states. This pipeline generates a precise, semantic-rich infrastructure inventory list.
• Figure 2: Architecture of the spatial attention-based matching module. Each 2D observation is encoded with geometric cues and passed through a Transformer to predict pairwise matchabilities. Auxiliary heads further regularize the representations by regressing 3D box size and location.
• Figure 3: Illustration of the geometry-guided matching refinement process. Note that for visualization clarity, the ray-to-center relationships are simplified in 2D, whereas our actual algorithm operates in 3D space. (Top) Initial State: Due to erroneous matching, ray $E$ (belonging to Target 2) is grouped with Target 1, causing a large geometric residual ($d > \tau_{split}$). (Middle) Step 1: Splitting. Ray $E$ is identified as a geometric outlier and pruned from the cluster, correcting the center of Target 1. (Bottom) Step 2: Merging. The isolated ray $E$ is re-evaluated and merged with ray $D$ based on geometric consistency ($d < \tau_{merge}$), successfully recovering Target 2.
• Figure 4: Workflow of the State-discriminative VLM Agent. The agent integrates Expert Knowledge Injection and Retrieval-Augmented Generation (RAG) to enhance a pre-trained VLM. Through Task-specific Multimodal Prompting with multi-view images, it accurately infers fine-grained attributes and operational states (e.g., "Obstructed"), delivering results in a structured JSON format.
• Figure 5: Visualization of object detection results on panoramic street-view images using the LoRA fine-tuned Grounding DINO model.