PIFu for the Real World: A Self-supervised Framework to Reconstruct Dressed Human from Single-view Images

TL;DR

This work proposes an end-to-end self-supervised network named SelfPIFu to utilize abundant and diverse in-the-wild images, resulting in largely improved reconstructions when tested on unconstrained in-the-wild images.

Abstract

It is very challenging to accurately reconstruct sophisticated human geometry caused by various poses and garments from a single image. Recently, works based on pixel-aligned implicit function (PIFu) have made a big step and achieved state-of-the-art fidelity on image-based 3D human digitization. However, the training of PIFu relies heavily on expensive and limited 3D ground truth data (i.e. synthetic data), thus hindering its generalization to more diverse real world images. In this work, we propose an end-to-end self-supervised network named SelfPIFu to utilize abundant and diverse in-the-wild images, resulting in largely improved reconstructions when tested on unconstrained in-the-wild images. At the core of SelfPIFu is the depth-guided volume-/surface-aware signed distance fields (SDF) learning, which enables self-supervised learning of a PIFu without access to GT mesh. The whole framework consists of a normal estimator, a depth estimator, and a SDF-based PIFu and better utilizes extra depth GT during training. Extensive experiments demonstrate the effectiveness of our self-supervised framework and the superiority of using depth as input. On synthetic data, our Intersection-Over-Union (IoU) achieves to 93.5%, 18% higher compared with PIFuHD. For in-the-wild images, we conduct user studies on the reconstructed results, the selection rate of our results is over 68% compared with other state-of-the-art methods.

Figures (11)

• Figure 1: Comparisons among different methods. PIFu takes a single color image as input (second column). PIFuHD takes both image and predicted-normal maps as input (third column). PaMIR, ICON, and ECON all utilizes SMPL prior (from fourth to sixth line), while ECON also introduces depth maps to optimize the coarse mesh. Ours takes a predicted-depth map as input (seventh column). The predcited-depth maps are shown in right. Ours contains more completed and reasonable geometry detailed.
• Figure 2: Overview of our framework. It consists of a normal estimator, a depth estimator, and a depth-guided SDF-based PIFu module that enables self-supervised learning. For synthetic data with 3D ground truth (top row) , we use fully-supervised learning method similar to PIFuHD saito2020pifuhd. For in-the-wild images without any labels (bottom row), we first estimate their depth maps, and then optimize the network parameters using a novel depth-guided self-supervised learning method (Sec. \ref{['sec:Self-V']}). We visualize the image and point-cloud-like depth map in the middle of frame.
• Figure 3: Depth-guided self-supervised learning.Volume-aware self-supervised SDF learning (left) utilizes on-/near-surface points converted from the estimated depth map to provide pseudo volume supervision. Surface-aware self-supervised SDF learning (right) imposes a surface-wise self-supervision by comparing DIST liu2020dist rendered and estimated depth maps. More details are described in Sec. \ref{['sec:Self-V']}.
• Figure 4: The structures of normal and depth estimator. All kernel size of convolutions set to 3*3.
• Figure 5: Comparison of using different input information as input, including image (denoted as "I"), normal map ("N"), depth map ("D"), and their combinations (e.g. "ID" for image and depth). Training (top) and test (bottom) loss curves during training are plotted.