ChatStitch: Visualizing Through Structures via Surround-View Unsupervised Deep Image Stitching with Collaborative LLM-Agents

TL;DR

ChatStitch addresses the challenge of surround-view perception in autonomous driving by combining a cognitively grounded closed-loop multi-agent framework with an unsupervised multi-image stitching module, SV-UDIS, designed for non-global-overlapping camera layouts. The system enables bidirectional, natural-language–driven perception refinement and integrates external 3D assets to reveal obscured objects, achieving SoTA results on UDIS-D and MCOV-SLAM datasets with significant PSNR and SSIM gains. SV-UDIS achieves robust stitching through three stages—masked cylindrical projection with feature extraction, motion propagation-based warping, and unsupervised mask-driven composition—enforced by rectangular warping constraints to reduce edge distortion. The work demonstrates the practical impact of human-in-the-loop perception for driving scenarios and lays a foundation for expanding perception capabilities, such as object detection and depth estimation, in future work.

Abstract

Surround-view perception has garnered significant attention for its ability to enhance the perception capabilities of autonomous driving vehicles through the exchange of information with surrounding cameras. However, existing surround-view perception systems are limited by inefficiencies in unidirectional interaction pattern with human and distortions in overlapping regions exponentially propagating into non-overlapping areas. To address these challenges, this paper introduces ChatStitch, a surround-view human-machine co-perception system capable of unveiling obscured blind spot information through natural language commands integrated with external digital assets. To dismantle the unidirectional interaction bottleneck, ChatStitch implements a cognitively grounded closed-loop interaction multi-agent framework based on Large Language Models. To suppress distortion propagation across overlapping boundaries, ChatStitch proposes SV-UDIS, a surround-view unsupervised deep image stitching method under the non-global-overlapping condition. We conducted extensive experiments on the UDIS-D, MCOV-SLAM open datasets, and our real-world dataset. Specifically, our SV-UDIS method achieves state-of-the-art performance on the UDIS-D dataset for 3, 4, and 5 image stitching tasks, with PSNR improvements of 9\%, 17\%, and 21\%, and SSIM improvements of 8\%, 18\%, and 26\%, respectively.

Figures (14)

• Figure 1: ChatStitch Reveals Occluded Vehicles in Stitched Surround-View Images via Language Commands.
• Figure 2: Schematic illustrations of different types of multi-image stitching tasks (specific definitions in \ref{['sec:sv-udis-problem-definition']}) and the effects of cylindrical projection.
• Figure 3: Overview of the ChatStitch System. ChatStitch breaks down complex human language into segments for different agents, each equipped with its own language processing model and executable functions.
• Figure 4: Overview of our proposed SV-UDIS. The pipeline mainly includes three stages: Masked cylindrical projection and feature extraction, multi-image warping, and multi-image composition. Our main contributions are shown in detail at the bottom of the figure.
• Figure 5: Result under a complex command. "With" and "without" represent the outcomes before and after our processing.