InpaintHuman: Reconstructing Occluded Humans with Multi-Scale UV Mapping and Identity-Preserving Diffusion Inpainting

TL;DR

InpaintHuman tackles reconstructing complete, animatable 3D human avatars from occluded monocular videos. It introduces a canonical pose-invariant representation based on multi-scale UV maps and Gaussian splats, plus an identity-preserving diffusion inpainting module that combines textual inversion for subject identity with ControlNet-based semantic guidance to ensure temporal coherence. Unlike SDS-based diffusion, it employs direct pixel-level supervision via a three-stage training pipeline with losses $L_{init}$, $L_{inpaint}$, and $L_{refine}$ to achieve consistent appearance across views and poses. Experiments on synthetic benchmarks (PeopleSnapshot, ZJU-MoCap) and real-world OcMotion demonstrate competitive or superior reconstruction and inpainting quality, validating robust occlusion handling and identity preservation. The approach advances practical avatar digitization under real-world occlusions for VR/AR and telepresence scenarios.

Abstract

Reconstructing complete and animatable 3D human avatars from monocular videos remains challenging, particularly under severe occlusions. While 3D Gaussian Splatting has enabled photorealistic human rendering, existing methods struggle with incomplete observations, often producing corrupted geometry and temporal inconsistencies. We present InpaintHuman, a novel method for generating high-fidelity, complete, and animatable avatars from occluded monocular videos. Our approach introduces two key innovations: (i) a multi-scale UV-parameterized representation with hierarchical coarse-to-fine feature interpolation, enabling robust reconstruction of occluded regions while preserving geometric details; and (ii) an identity-preserving diffusion inpainting module that integrates textual inversion with semantic-conditioned guidance for subject-specific, temporally coherent completion. Unlike SDS-based methods, our approach employs direct pixel-level supervision to ensure identity fidelity. Experiments on synthetic benchmarks (PeopleSnapshot, ZJU-MoCap) and real-world scenarios (OcMotion) demonstrate competitive performance with consistent improvements in reconstruction quality across diverse poses and viewpoints.

Figures (6)

• Figure 1: Given a video with significant occlusions (a), existing methods produce incomplete or inconsistent reconstructions (c,d). InpaintHuman leverages occlusion-robust multi-scale UV-parameterized representation and identity-preserving diffusion inpainting to reconstruct a complete, animatable avatar with consistent appearance across novel views and poses (b,e).
• Figure 2: Overview of the InpaintHuman.(a) 3D Human Rendering: We represent the human avatar using 3D Gaussians anchored on the SMPL mesh, with attributes predicted from multi-scale UV feature maps that enable robust interpolation across occluded regions. These Gaussians are transformed to observation space via forward LBS, augmented with pose-dependent residual features for non-rigid dynamics. (b) Identity-Preserving Diffusion Inpainting: A personalized Stable Diffusion inpainting model takes occluded images and visibility masks as input. Subject-level identity is captured via textual inversion with a learnable token, while pose consistency is ensured through ControlNet-based semantic guidance. (c) Refinement: Inpainted images supervise the optimization of canonical UV maps, propagating plausible content to occluded regions and yielding a complete, animatable avatar.
• Figure 3: Multi-scale UV feature maps for occlusion robustness. Coarser resolutions (e.g., $64 \times 64$) compress spatial distances between visible and occluded regions, facilitating feature interpolation but lacking fine details. Higher resolutions (e.g., $256 \times 256$) preserve geometric details but are more susceptible to incomplete observations. Our hierarchical design combines both advantages: robust occlusion handling with high-fidelity detail preservation.
• Figure 4: Qualitative comparison on novel view synthesis. We present results on ZJU-MoCap peng2021neural with synthetic occlusions (left) and OcMotion huang2022occluded with real-world occlusions (right). OccNeRF xiang2023rendering struggles to hallucinate unseen regions, often producing noticeable discoloration. OccFusion sun2024occfusion generates sharper textures in some areas but exhibits blurriness and visual uncertainty in heavily occluded regions. Our method produces more complete renderings with better preservation of subject-specific appearance.
• Figure 5: Qualitative comparison of inpainting results. Given occluded input images (a), we compare completions from OccFusion sun2024occfusion (b), GTU lee2024guess (c), and our method (d), with ground truth reference (e). Our identity-preserving diffusion module generates textures that maintain appearance consistency with visible regions and spatial plausibility respecting body structure. In contrast, SDS-based methods (b, c) exhibit identity drift with inconsistent colors and patterns.