GeoComplete: Geometry-Aware Diffusion for Reference-Driven Image Completion

TL;DR

GeoComplete tackles reference-driven image completion when the target view diverges from references by injecting explicit 3D geometry into diffusion and guiding cue selection with target-aware masking. It combines a geometry-conditioned dual-branch diffusion with a joint self-attention mechanism to fuse visual and geometric cues, aided by VGGT for camera/depth estimation and LangSAM for dynamic object filtering. The method demonstrates substantial gains in PSNR and perceptual metrics on RealBench and QualBench, with quantified improvements over strong baselines, and shows robustness to upstream localization and segmentation errors. This geometry-aware approach reduces hallucinations and improves spatial consistency, enabling more reliable scene completion in challenging multi-view scenarios.

Abstract

Reference-driven image completion, which restores missing regions in a target view using additional images, is particularly challenging when the target view differs significantly from the references. Existing generative methods rely solely on diffusion priors and, without geometric cues such as camera pose or depth, often produce misaligned or implausible content. We propose GeoComplete, a novel framework that incorporates explicit 3D structural guidance to enforce geometric consistency in the completed regions, setting it apart from prior image-only approaches. GeoComplete introduces two key ideas: conditioning the diffusion process on projected point clouds to infuse geometric information, and applying target-aware masking to guide the model toward relevant reference cues. The framework features a dual-branch diffusion architecture. One branch synthesizes the missing regions from the masked target, while the other extracts geometric features from the projected point cloud. Joint self-attention across branches ensures coherent and accurate completion. To address regions visible in references but absent in the target, we project the target view into each reference to detect occluded areas, which are then masked during training. This target-aware masking directs the model to focus on useful cues, enhancing performance in difficult scenarios. By integrating a geometry-aware dual-branch diffusion architecture with a target-aware masking strategy, GeoComplete offers a unified and robust solution for geometry-conditioned image completion. Experiments show that GeoComplete achieves a 17.1 PSNR improvement over state-of-the-art methods, significantly boosting geometric accuracy while maintaining high visual quality.

Figures (5)

• Figure 1: Given a few reference images of the same scene and a target image with missing regions, our method completes the target's missing regions while preserving geometric consistency more effectively than the state-of-the-art Paint-by-Example yang2023paint. Semi-transparent white masks indicate the known, unaltered regions of the target image.
• Figure 2: Overview of our GeoComplete framework: We first construct a point cloud from the reference and target images. During training, target-aware masking selectively occludes both reference images and their projected point clouds to highlight informative regions. These inputs are processed by a dual-branch diffusion model: the target branch encodes the masked image, while the cloud branch encodes the projected point cloud. Joint self-attention fuses the two, allowing geometric cues to guide synthesis. At inference, the masked target image and its projected point cloud are fed into the finetuned model to complete the missing regions.
• Figure 3: Overview of our point cloud generation pipeline. Given reference and target images, we first obtain a text prompt describing dynamic objects, either provided by users or generated by an LLM achiam2023gpt. Based on the prompt, LangSAM medeiros2023langsamliu2024groundingravi2024sam is employed to segment and remove dynamic regions. VGGT wang2025vggt is then applied to estimate camera parameters and depth maps, which are used to construct and project the 3D point cloud.
• Figure 4: Qualitative comparisons from Transfill zhou2021transfill, RealFill tang2024realfill, Paint-by-Example yang2023paint and our method. The red bounding box marks the known, unaltered region of the target image (i.e., the area inside the box), except for the first-row images, where the known region lies outside the box. Our method synthesizes missing regions while ensuring better geometric consistency.
• Figure 5: Qualitative comparisons from Transfill zhou2021transfill, RealFill tang2024realfill, Paint-by-Example yang2023paint and our method. The red bounding box marks the known, unaltered region of the target image (i.e., the area inside the box). Our method synthesizes missing regions while ensuring better geometric consistency.