3D Human Pose Estimation Based on 2D-3D Consistency with Synchronized Adversarial Training

TL;DR

The paper tackles depth ambiguity in monocular 3D human pose estimation by proposing a synchronized adversarial framework that jointly teaches a generator, a reprojection network, and a discriminator to enforce 2D-3D consistency. A weighted KCS matrix is incorporated into the discriminator to enforce bone-length and joint-angle constraints, while a reprojection network learns the distribution mapping from 3D poses back to 2D reprojections. Empirical results on Human3.6M, MPI-INF-3DHP, and MPII show state-of-the-art performance among weakly supervised methods, with robust handling of 2D detector noise and strong generalization to in-the-wild data. The approach advances practical 3D pose estimation by reducing dependency on large 3D labels and improving geometric consistency across views and poses.

Abstract

3D human pose estimation from a single image is still a challenging problem despite the large amount of work that has been performed in this field. Generally, most methods directly use neural networks and ignore certain constraints (e.g., reprojection constraints, joint angle, and bone length constraints). While a few methods consider these constraints but train the network separately, they cannot effectively solve the depth ambiguity problem. In this paper, we propose a GAN-based model for 3D human pose estimation, in which a reprojection network is employed to learn the mapping of the distribution from 3D poses to 2D poses, and a discriminator is employed for 2D-3D consistency discrimination. We adopt a novel strategy to synchronously train the generator, the reprojection network and the discriminator. Furthermore, inspired by the typical kinematic chain space (KCS) matrix, we introduce a weighted KCS matrix and take it as one of the discriminator's inputs to impose joint angle and bone length constraints. The experimental results on Human3.6M show that our method significantly outperforms state-of-the-art methods in most cases.

Figures (8)

• Figure 1: Overview of our model. We feed the estimated 3D pose and 2D reprojection to the discriminator simultaneously, and train the generator, reprojection network, and discriminator synchronously.
• Figure 2: The main structure of our proposed adversarial training framework, which contains 4 parts: (1) generator, (2) discriminator, (3) reprojection network, and (4) three loss functions. Our structure takes 2D reprojected poses and 3D generated poses into consideration simultaneously. In pratice, the generator, discriminator and reprojection network will be trained synchronously.
• Figure 3: The structure of our generator and reprojection network is roughly the same as Martinez2017A. The generator has two residual blocks, and the reprojection network has only one residual block. The generator's input is a 2D pose, and the output is the corresponding $z$-direction component. In contrast, the reprojection network's input is a 3D pose, and the output is the related reprojected 2D pose.
• Figure 4: The structure of the discriminator.
• Figure 5: The bone distances $d_{ij}$ from the right hip bone.