End-to-end Recovery of Human Shape and Pose

TL;DR

This work introduces HMR, an end-to-end framework that recovers a full 3D human mesh from a single RGB image by regressing SMPL shape and pose parameters. It combines reprojection losses with a factorized adversarial prior to constrain outputs to the manifold of plausible human bodies, enabling training with or without paired 3D supervision. The model uses iterative error feedback to directly predict SMPL parameters from image features and benefits from a diverse, unpaired set of 3D meshes as a weak global prior. Empirically, HMR achieves strong 3D joint estimation and segmentation performance in-the-wild and remains robust when 3D supervision is unavailable, while running in real time.

Abstract

We describe Human Mesh Recovery (HMR), an end-to-end framework for reconstructing a full 3D mesh of a human body from a single RGB image. In contrast to most current methods that compute 2D or 3D joint locations, we produce a richer and more useful mesh representation that is parameterized by shape and 3D joint angles. The main objective is to minimize the reprojection loss of keypoints, which allow our model to be trained using images in-the-wild that only have ground truth 2D annotations. However, the reprojection loss alone leaves the model highly under constrained. In this work we address this problem by introducing an adversary trained to tell whether a human body parameter is real or not using a large database of 3D human meshes. We show that HMR can be trained with and without using any paired 2D-to-3D supervision. We do not rely on intermediate 2D keypoint detections and infer 3D pose and shape parameters directly from image pixels. Our model runs in real-time given a bounding box containing the person. We demonstrate our approach on various images in-the-wild and out-perform previous optimization based methods that output 3D meshes and show competitive results on tasks such as 3D joint location estimation and part segmentation.

Figures (5)

• Figure 1: Human Mesh Recovery (HMR): End-to-end adversarial learning of human pose and shape. We describe a real time framework for recovering the 3D joint angles and shape of the body from a single RGB image. The first two rows show results from our model trained with some 2D-to-3D supervision, the bottom row shows results from a model that is trained in a fully weakly-supervised manner without using any paired 2D-to-3D supervision. We infer the full 3D body even in case of occlusions and truncations. Note that we capture head and limb orientations.
• Figure 2: Overview of the proposed framework. An image $I$ is passed through a convolutional encoder. This is sent to an iterative 3D regression module that infers the latent 3D representation of the human that minimizes the joint reprojection error. The 3D parameters are also sent to the discriminator $D$, whose goal is to tell if these parameters come from a real human shape and pose.
• Figure 3: Results sampled from different datasets at the 15th, 30th, 60th, 90th and 95th error percentiles. Percentiles are computed using MPJPE for 3D datasets (first two rows - Human3.6M and MPI-INF-3DHP) and 2D pose PCK for 2D datasets (last two rows - LSP and MS COCO). High percentile indicates high error. Note results at high error percentile are often semantically quite reasonable.
• Figure 4: Results with and without paired 3D supervision. 3D reconstructions, without direct 3D supervision, are very close to those of the supervised model.
• Figure 5: No Discriminator No 3D. With neither the discriminator, nor the direct 3D supervision, the network produces monsters. On the right of each example we visualize the ground truth keypoint annotation in unfilled circles, and the projection in filled circles. Note that despite the unnatural pose and shape, its 2D projection error is very accurate.