Flexible Geometric Guidance for Probabilistic Human Pose Estimation with Diffusion Models

TL;DR

The paper tackles the inherent depth ambiguity in monocular 3D human pose estimation by proposing a probabilistic framework based on diffusion models guided by 2D keypoint heatmaps. It trains an unconditional 3D pose prior and uses geometric guidance to sample conditional poses without needing paired 2D-3D data, enabling flexible conditioning and controllable diversity. The approach achieves state-of-the-art performance among correspondence-free probabilistic methods on Human 3.6M and competitive results on MPI-INF-3DHP and 3DPW, while enabling tasks such as pose completion and unconditional generation. The work provides code and highlights ethical considerations for biometric data use and bias mitigation.

Abstract

3D human pose estimation from 2D images is a challenging problem due to depth ambiguity and occlusion. Because of these challenges the task is underdetermined, where there exists multiple -- possibly infinite -- poses that are plausible given the image. Despite this, many prior works assume the existence of a deterministic mapping and estimate a single pose given an image. Furthermore, methods based on machine learning require a large amount of paired 2D-3D data to train and suffer from generalization issues to unseen scenarios. To address both of these issues, we propose a framework for pose estimation using diffusion models, which enables sampling from a probability distribution over plausible poses which are consistent with a 2D image. Our approach falls under the guidance framework for conditional generation, and guides samples from an unconditional diffusion model, trained only on 3D data, using the gradients of the heatmaps from a 2D keypoint detector. We evaluate our method on the Human 3.6M dataset under best-of-$m$ multiple hypothesis evaluation, showing state-of-the-art performance among methods which do not require paired 2D-3D data for training. We additionally evaluate the generalization ability using the MPI-INF-3DHP and 3DPW datasets and demonstrate competitive performance. Finally, we demonstrate the flexibility of our framework by using it for novel tasks including pose generation and pose completion, without the need to train bespoke conditional models. We make code available at https://github.com/fsnelgar/diffusion_pose .

Figures (15)

• Figure 1: Overview of our pose estimation method. Using the 2D detections (modeled as Gaussians) from a keypoint detector $f_\phi$, we guide the reverse process of an unconditional diffusion model $p_\theta$ to sample 3D poses from a conditional distribution using geometric guidance. Only 3D poses are required for training.
• Figure 2: MPJPE as a function of number of samples for Human 3.6M and 3DPW datasets. Results for ZeDO Jiang2024ZeDO were reproduced from the official repository.
• Figure 3: Qualitative examples of diverse 3D human poses generated by our method without guidance.
• Figure 4: Qualitative examples of pose completion results. Observation likelihood is not defined for red joints and the model must 'inpaint' these joints. Grayscale images illustrated for context.
• Figure 5: Qualitative examples of the effect of scaling covariance matrices $\Sigma^{(j)}$. Each column has the same latent variable; the scaling factor decreases down the column.