InfraGPT Smart Infrastructure: An End-to-End VLM-Based Framework for Detecting and Managing Urban Defects

TL;DR

InfraGPT addresses the gap between perception and maintenance decision making in smart-city infrastructure by turning CCTV imagery into structured maintenance plans. It combines a YOLO-based multi-defect detector with a vision-language reasoning component to ground detections in context and output a JSON plan that includes incident type, location, confidence, required tools, urgency, and actions. End-to-end experiments on public datasets and captured CCTV clips show strong defect detection performance and coherent, schema-conformant maintenance summaries, demonstrating effective perception–reasoning integration. The work enables scalable, data-driven infrastructure management with potential for GIS and digital twin integration, while discussing challenges in latency, domain shifts, and schema alignment for city-wide deployment.

Abstract

Infrastructure in smart cities is increasingly monitored by networks of closed circuit television (CCTV) cameras. Roads, bridges and tunnels develop cracks, potholes, and fluid leaks that threaten public safety and require timely repair. Manual inspection is costly and hazardous, and existing automatic systems typically address individual defect types or provide unstructured outputs that cannot directly guide maintenance crews. This paper proposes a comprehensive pipeline that leverages street CCTV streams for multi defect detection and segmentation using the YOLO family of object detectors and passes the detections to a vision language model (VLM) for scene aware summarization. The VLM generates a structured action plan in JSON format that includes incident descriptions, recommended tools, dimensions, repair plans, and urgent alerts. We review literature on pothole, crack and leak detection, highlight recent advances in large vision language models such as QwenVL and LLaVA, and describe the design of our early prototype. Experimental evaluation on public datasets and captured CCTV clips demonstrates that the system accurately identifies diverse defects and produces coherent summaries. We conclude by discussing challenges and directions for scaling the system to city wide deployments.

Figures (4)

• Figure 1: Overview of the InfraGPT architecture. The system processes visual inputs from various camera sources through a Vision--Language Model (VLM) that performs initial scene screening for defects such as cracks or leaks. According to the detected context, the VLM selects the appropriate YOLO model for precise localization and classification. The resulting detections are reanalyzed by the VLM to produce a structured JSON-based maintenance plan that includes incident details, required tools, and urgency levels. This end-to-end design enables adaptive perception and automated decision generation for infrastructure monitoring.
• Figure 2: Comprehensive Vision--Language analysis: (a) per-image ROUGE-L F1 dumbbell plot; (b) parity plot of predicted vs. reference scores; (c) macro-level ROUGE-L F1/P/R and $\Delta$ROUGE-L analysis.
• Figure 3: YOLO validation metrics showing Precision, Classification Loss, and mAP evolution over epochs. The curves reflect performance on the held-out validation set, indicating convergence and generalization across defect categories.
• Figure 4: Example results from the combined YOLO--VLM pipeline showing input image, detections, and generated structured maintenance plans. The VLM translates localized detections into actionable maintenance instructions.