CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image

TL;DR

CHROME tackles the problem of reconstructing occluded clothed humans in a multiview-consistent manner from a single image without relying on ground-truth 3D supervision or SMPL priors. It introduces a two-stage approach: a pose-controlled multiview diffusion model (F_D) that generates four occlusion-free, cross-view-consistent views from the occluded input, and a 3D reconstruction module (F_R) that fuses the original image with the synthesized views into a cohesive 3D Gaussian representation comprised of $n_g$ Gaussians, each described by a 14D parameter vector. The framework uses differentiable rendering and a combination of MSE, LPIPS, and silhouette losses to supervise the 3D Gaussian field, enabling occlusion-resilient novel view synthesis and accurate geometry. The results show CHROME achieves state-of-the-art occlusion robustness and multiview coherence on multiple datasets, including THuman2.0, CustomHumans, CAPE, and AHP, and extends naturally to stereo input scenarios with favorable inference times, suggesting strong practical impact for VR/AR, digital avatars, and fashion apps.

Abstract

Reconstructing clothed humans from a single image is a fundamental task in computer vision with wide-ranging applications. Although existing monocular clothed human reconstruction solutions have shown promising results, they often rely on the assumption that the human subject is in an occlusion-free environment. Thus, when encountering in-the-wild occluded images, these algorithms produce multiview inconsistent and fragmented reconstructions. Additionally, most algorithms for monocular 3D human reconstruction leverage geometric priors such as SMPL annotations for training and inference, which are extremely challenging to acquire in real-world applications. To address these limitations, we propose CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-ConsistEncy from a Single Image, a novel pipeline designed to reconstruct occlusion-resilient 3D humans with multiview consistency from a single occluded image, without requiring either ground-truth geometric prior annotations or 3D supervision. Specifically, CHROME leverages a multiview diffusion model to first synthesize occlusion-free human images from the occluded input, compatible with off-the-shelf pose control to explicitly enforce cross-view consistency during synthesis. A 3D reconstruction model is then trained to predict a set of 3D Gaussians conditioned on both the occluded input and synthesized views, aligning cross-view details to produce a cohesive and accurate 3D representation. CHROME achieves significant improvements in terms of both novel view synthesis (upto 3 db PSNR) and geometric reconstruction under challenging conditions.

Figures (20)

• Figure 1: Need for occlusion-resilient multiview-consistent clothed human reconstruction from single images: Existing algorithms for monocular 3D clothed human reconstruction, such as PIFu saito2019pifu and SIFU zhang2024sifu, produce fragmented and multiview-inconsistent novel view reconstructions from single-view occluded human images zhang2019pose2seg. In contrast, CHROME generates occlusion-free, multiview-consistent novel views from a single occluded image, using a novel pose-controlled multiview diffusion model, combined with a large reconstruction model, to create a cohesive 3D representation of the human subject. Moreover, CHROME does not require 3D mesh supervision during training as required for existing algorithms saito2019pifuzhang2024sifu.
• Figure 2: Overview of proposed method:$\texttt{CHROME}$ is a novel two-stage pipeline designed for occlusion-free 3D human reconstruction from a single occluded image. In the first stage, a pose-controlled multiview diffusion model generates four de-occluded views of the subject, ensuring pose consistency across the synthesized images, conditioned on the input occluded image and the 3D pose estimates. In the second stage, a 3D reconstruction model combines the occluded input image with the synthesized multi-view images to create a cohesive 3D Gaussian representation, aligning geometric and texture details across views, enabling accurate reconstructions and robust novel view synthesis, even in the presence of significant occlusions. Note that the method is also compatible with multi-view inputs.
• Figure 3: Qualitative analysis of $\mathcal{F}_{D}$ in generating occlusion-free novel views: Qualitative results demonstrating the capacity of $\mathcal{F}_{D}$ in generating occlusion-free novel views conditioned on a single occluded image from CustomHumans.
• Figure 4: Qualitative analysis of $\texttt{CHROME}$ on artificially occluded THuman2.0 and CustomHumans: Qualitative results of $\texttt{CHROME}$ against state-of-the-art algorithms saito2019pifuzhang2024globalzhang2024sifuho2024sith on artificially occluded THuman2.0 (top three rows) and artificially occluded CustomHumans (bottom three rows); clearly none of the baseline algorithm perform occlusion-free novel view synthesis whereas $\texttt{CHROME}$ effectively mitigates occlusions and predicts multiview consistent reconstructions. Additional qualitative results are provided in the supplementary.
• Figure 5: Qualitative analysis of geometric reconstruction via normal maps: Qualitative comparisons of $\texttt{CHROME}$ against state-of-the-art methods for geometric reconstruction via normal consistency. Clearly, the predictions from existing algorithms are not occlusion-resilient, generating implausible reconstructions whereas $\texttt{CHROME}$ effectively handles occlusions, producing occlusion-resilient reconstructions.