IntegratedPIFu: Integrated Pixel Aligned Implicit Function for Single-view Human Reconstruction

TL;DR

IntegratedPIFu shows how depth and human parsing information can be predicted and capitalised upon in a pixel-aligned implicit model and introduces depth oriented sampling, a novel training scheme that improve any pixel aligned implicit model ability to reconstruct important human features without noisy artefacts.

Abstract

We propose IntegratedPIFu, a new pixel aligned implicit model that builds on the foundation set by PIFuHD. IntegratedPIFu shows how depth and human parsing information can be predicted and capitalised upon in a pixel-aligned implicit model. In addition, IntegratedPIFu introduces depth oriented sampling, a novel training scheme that improve any pixel aligned implicit model ability to reconstruct important human features without noisy artefacts. Lastly, IntegratedPIFu presents a new architecture that, despite using less model parameters than PIFuHD, is able to improves the structural correctness of reconstructed meshes. Our results show that IntegratedPIFu significantly outperforms existing state of the arts methods on single view human reconstruction. Our code has been made available online.

Figures (7)

• Figure 1: Compared with state-of-the-art methods, including (b) Geo-PIFu he2020geo, (c) PIFu saito2019pifu, and (d) PIFuHD saito2020pifuhd, our proposed model can precisely reconstruct various parts of a clothed human body without noisy artefacts. Subject's face censored as required by dataset's owner
• Figure 2: Overview of our IntegratedPIFu framework. The Low-Resolution PIFu produces coarse feature maps that are further refined by High-Resolution Integrator. The High-Resolution Integrator will then use a MLP to generate a reconstructed mesh. *We recommend the use of predicted depth and human parsing maps in the Low-Resolution PIFu, but the use of them in the High-Resolution Integrator is optional and depends heavily on the quality of the two maps.
• Figure 3: Illustration of Depth-Oriented Sampling. The red dots represent points that are exactly on the mesh surface. These red dots are then randomly displaced. The blue dots represent the possible locations of the red dots after displacement
• Figure 4: Qualitative evaluation with SOTA methods, including (b) Geo-PIFu he2020geo, (c) PIFu saito2019pifu, and (d) PIFuHD saito2020pifuhd. The input RGB image is shown as the first object in each row. For each method, we show the frontal view and an alternative view. For the last row, the reconstructed meshes are colored by projecting the RGB values from the input image onto the generated meshes. The coloring serves as visual aids only.
• Figure 5: Comparison with different backbones. (b) is a vanilla PIFu, (c) is given predicted depth map as additional input, (d) is given predicted human parsing, (e) is given both maps. To aid visualization, some reconstructed meshes are colored by projecting the RGB values from the input image onto the generated meshes