X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimation

TL;DR

This work proposes a novel unsupervised framework featuring a multi-hypothesis detector and multiple tailored pretext tasks to tackle depth ambiguity and demonstrates state-of-the-art unsupervised 3D pose estimation performance on various human datasets.

Abstract

Recent unsupervised methods for monocular 3D pose estimation have endeavored to reduce dependence on limited annotated 3D data, but most are solely formulated in 2D space, overlooking the inherent depth ambiguity issue. Due to the information loss in 3D-to-2D projection, multiple potential depths may exist, yet only some of them are plausible in human structure. To tackle depth ambiguity, we propose a novel unsupervised framework featuring a multi-hypothesis detector and multiple tailored pretext tasks. The detector extracts multiple hypotheses from a heatmap within a local window, effectively managing the multi-solution problem. Furthermore, the pretext tasks harness 3D human priors from the SMPL model to regularize the solution space of pose estimation, aligning it with the empirical distribution of 3D human structures. This regularization is partially achieved through a GCN-based discriminator within the discriminative learning, and is further complemented with synthetic images through rendering, ensuring plausible estimations. Consequently, our approach demonstrates state-of-the-art unsupervised 3D pose estimation performance on various human datasets. Further evaluations on data scale-up and one animal dataset highlight its generalization capabilities. Code will be available at https://github.com/Charrrrrlie/X-as-Supervision.

Figures (10)

• Figure 1: Challenges. (a) illustrates the pretext tasks of related methods, which localize keypoints by various structure representations for human reconstruction. However, they are limited in 2D space without direct depth supervision. (b) highlights the widely observed depth ambiguity issue, where similar 2D poses correspond to distinctive relative depths from the pelvis and not all 3D correspondences along the ray of light are plausible for human structure. It remains an unexplored constraint problem in unsupervised monocular 3D pose estimation.
• Figure 1: Comparison with state-of-the-art methods on Human3.6M. As discussed in \ref{['sec:related-work']}: SPP: supervised post-processing. RI/MV: reference image or multi-view. T: template. J: unpaired ground truth joints. Our baselines are introduced in \ref{['sec:our-baseline']}. MPJPEs are in $mm$.
• Figure 2: Framework overview of the X as Supervision. It contains a multi-hypothesis detector and novel pretext tasks for unsupervised training. The proposed detector takes in monocular images and novelly decodes multiple depth solutions by aggregating the local responses of the heatmap. For pretext tasks, in the 2D space, we adopt Masks as Supervision yang2023mask for keypoints localization and human structure. In the 3D space, we introduce human priors as constraints from a curated SMPL set via discriminative learning and synthetic pairs.
• Figure 3: Visualization of the predicted multiple hypotheses on Human3.6M ionescu2013human3 and MPI-INF-3DHP mono-3dhp2017. Differences are highlighted in red circles. The ground truth poses are plotted in the last column. More visualization results are presented in \ref{['sec:supp-qualitative']}.
• Figure 4: Illustration of different textures. From top to down: SURREAL varol2017learning human texture, part segmentation of SMPL loper2023smpl and SMAL Zuffi2017smal.