FAMOUS: High-Fidelity Monocular 3D Human Digitization Using View Synthesis

TL;DR

FAMOUS tackles the texture gap in monocular 3D human digitization by leveraging abundant 2D fashion data to hallucinate back-view textures through a distributionally aligned hallucinator. A disentangled domain alignment strategy (SA and PA) progressively adapts the hallucinator from 2D fashion distributions to target 3D datasets, enabling improved texture synthesis that complements occupancy-based geometry prediction. The framework fuses frontal input with synthesized back-view information to produce high-fidelity textured meshes, validated against RenderPeople and DeepFashion-derived baselines, and outperforming several state-of-the-art texture-and-geometry methods. This approach offers scalable texture realism for AR/VR/MR avatars and demonstrates the viability of cross-domain 2D priors to enrich 3D reconstructions, with public code and data supplements to foster further research.

Abstract

The advancement in deep implicit modeling and articulated models has significantly enhanced the process of digitizing human figures in 3D from just a single image. While state-of-the-art methods have greatly improved geometric precision, the challenge of accurately inferring texture remains, particularly in obscured areas such as the back of a person in frontal-view images. This limitation in texture prediction largely stems from the scarcity of large-scale and diverse 3D datasets, whereas their 2D counterparts are abundant and easily accessible. To address this issue, our paper proposes leveraging extensive 2D fashion datasets to enhance both texture and shape prediction in 3D human digitization. We incorporate 2D priors from the fashion dataset to learn the occluded back view, refined with our proposed domain alignment strategy. We then fuse this information with the input image to obtain a fully textured mesh of the given person. Through extensive experimentation on standard 3D human benchmarks, we demonstrate the superior performance of our approach in terms of both texture and geometry. Code and dataset is available at https://github.com/humansensinglab/FAMOUS.

Figures (7)

• Figure 1: Visual comparison between monocular 3D human digitization methods. Given a single image input, we demonstrate the renderings of textured mesh generated by our approach (FAMOUS) and SOTA pipelines (i.e. predicting both the shape and texture): PIFu pifu, PaMIR pamir, and PHORHUM phorhum. Our approach presents significantly improved results in terms of texture and geometry.
• Figure 2: Method overview. Given a single frontal image of a subject, FAMOUS generates a high-quality textured mesh. Our distributionally aligned hallucinator (Section \ref{['subsec:dah']}) first predicts the back view based on the reference image, SMPL-X segmentation map and silhouette. Next, the normal map generator $\{(\hat{G}^\mathcal{N}_f, \hat{G}^\mathcal{N}_b)\}$ takes the reference image and generated back image as input and output the respective normal maps. The reference image and the normal estimates are leveraged by the implicit function network (IF) for the geometry prediction (Section \ref{['subsec:occ']}). Finally, our learnable texture prediction module refines the 3D texture aggregated from the input reference image and predicted back view (Section \ref{['subsec:tex']}).
• Figure 3: Semantic Alignment (SA): Given a partial image $I^s$ and full body COCO skeleton ${(X_{f}, X_{b})}$ from our 2D fashion dataset (i.e. source), we generate the front/back pairs ${(I^\mathcal{SA}_f, I^\mathcal{SA}_b )}$ similar to our full body target 3D dataset (thus aligning the semantics) with a frozen hallucinator ($G^s$). Next, we finetune the hallucinator on these full body pseudo pairs from one another. The hallucinator is initialized with pretrained weights from nted, indicated by superscript $s$. The weight sharing in each stage is indicated by the bidirectional dotted arrows. The default loss function ($\mathcal{L}$) proposed in nted is used.
• Figure 4: Dataset: The sets of images randomly sampled from the source dataset (top) with their corresponding SA (mid) and PA (bottom) pairs.
• Figure 5: Pose Alignment (PA):Given the subset of the pseudo pairs ${ ( I^\mathcal{SA}_f, I^\mathcal{SA}_b ) }$ after SA, we generate the front/back pairs following the pose from the target distribution ${ ( I^\mathcal{PA}_f, I^\mathcal{PA}_b ) }$ to finetune the semantically aligned hallucinator further. The hallucinator is initialized with weights obtained after SA.