An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface Reconstruction

TL;DR

The paper addresses reconstructing 3D clothed human surfaces from a single image while preserving SMPL pose/shape priors. It introduces the IUVD-Feedback representation, embedding SMPL UV maps into an IUVD space and replacing costly SDF-based queries with a linear transformation, augmented by a feedback-driven query strategy. Experiments on THuman2.0 demonstrate threefold acceleration in the query-and-infer stage and improved robustness, without retraining networks, with added potential for part-based editing and generative applications due to the semantic IUVD structure. The approach bridges SMPL-based priors and implicit surface reconstruction, delivering faster, more stable, and semantically rich clothed human reconstructions suitable for integration into existing pipelines and downstream tasks.

Abstract

To reconstruct a 3D human surface from a single image, it is crucial to simultaneously consider human pose, shape, and clothing details. Recent approaches have combined parametric body models (such as SMPL), which capture body pose and shape priors, with neural implicit functions that flexibly learn clothing details. However, this combined representation introduces additional computation, e.g. signed distance calculation in 3D body feature extraction, leading to redundancy in the implicit query-and-infer process and failing to preserve the underlying body shape prior. To address these issues, we propose a novel IUVD-Feedback representation, consisting of an IUVD occupancy function and a feedback query algorithm. This representation replaces the time-consuming signed distance calculation with a simple linear transformation in the IUVD space, leveraging the SMPL UV maps. Additionally, it reduces redundant query points through a feedback mechanism, leading to more reasonable 3D body features and more effective query points, thereby preserving the parametric body prior. Moreover, the IUVD-Feedback representation can be embedded into any existing implicit human reconstruction pipeline without requiring modifications to the trained neural networks. Experiments on the THuman2.0 dataset demonstrate that the proposed IUVD-Feedback representation improves the robustness of results and achieves three times faster acceleration in the query-and-infer process. Furthermore, this representation holds potential for generative applications by leveraging its inherent semantic information from the parametric body model.

Figures (15)

• Figure 1: Comparing the pipelines of implicit human surface reconstruction in the traditional XYZ space and in the proposed IUVD space, there are three advantages of the proposed IUVD-Feedback representation: 1) It accelerates the query&infer and visualization steps, by replacing the redundant SDF calculation and reducing the complexity of marching cubes respectively. 2) It preserves more robust topology of human surface, thus preventing non-human shapes. 3) It produces semantic-aware results, which enables part-based surface editing. Note that the encoders and implicit function modules are replaceable and the space encoders are optional in the pipeline.
• Figure 2: Overview of 3D human surface reconstruction with the proposed IUVD-Feedback representation. Given a masked color image, we first obtain the SMPL mesh, body normal maps and cloth normal maps where HPS denotes Human Pose and Shape estimation and DR denotes Differentiable Rendering. Then the implicit 3D human surface reconstruction process xiu2022icon is restructured and accelerated in IUVD space (Sec. \ref{['sec:iuvd_representation']}) by three steps: 1) XYZ to IUVD feature space transformation (Sec. \ref{['sec:iuvd_feature']}), 2) IUVD occupancy inference with feedback query (Sec. \ref{['sec:iuvd_query']}), and 3) IUVD to XYZ mesh transformation (Sec. \ref{['sec:iuvd_mesh']}). Finally, the part-based meshes are combined and optionally refined in XYZ space as the reconstruction result.
• Figure 3: Scaling of the SMPL UV maps for nearly uniform sampling. The color indicates the proportions of different body parts. The black area is obsolete.
• Figure 4: By changing the source of query points, the local body features in XYZ space are transformed to IUVD space. And the time-consuming P2S optimization (red) is replaced by a simple linear transformation (green) for constructing the IUVD local body features.
• Figure 5: For a query point $P$, the corresponding source point $P^b$ is unique when the surface is convex. But there may exist multiple source points, e.g. $P^b_1$, $P^b_2$, when the surface is concave.