Flow Matching for Probabilistic Monocular 3D Human Pose Estimation

TL;DR

The paper addresses the challenge of monocular 3D human pose estimation by treating 3D poses as a distribution rather than a single estimate. It introduces FMPose, a probabilistic method that uses flow matching with continuous normalizing flows and optimal transport to map a simple Gaussian to the posterior 3D pose distribution, conditioned on 2D heatmap cues via a learnable GCN-based lifting condition. It demonstrates state-of-the-art performance on Human3.6M, MPI-INF-3DHP, and 3DPW, while offering faster and more stable generation than diffusion-based approaches. By providing a rich set of pose hypotheses with calibrated uncertainty, FMPose improves robustness in occluded or highly ambiguous scenarios and learns meaningful joint relations through the adjacency matrix.

Abstract

Recovering 3D human poses from a monocular camera view is a highly ill-posed problem due to the depth ambiguity. Earlier studies on 3D human pose lifting from 2D often contain incorrect-yet-overconfident 3D estimations. To mitigate the problem, emerging probabilistic approaches treat the 3D estimations as a distribution, taking into account the uncertainty measurement of the poses. Falling in a similar category, we proposed FMPose, a probabilistic 3D human pose estimation method based on the flow matching generative approach. Conditioned on the 2D cues, the flow matching scheme learns the optimal transport from a simple source distribution to the plausible 3D human pose distribution via continuous normalizing flows. The 2D lifting condition is modeled via graph convolutional networks, leveraging the learnable connections between human body joints as the graph structure for feature aggregation. Compared to diffusion-based methods, the FMPose with optimal transport produces faster and more accurate 3D pose generations. Experimental results show major improvements of our FMPose over current state-of-the-art methods on three common benchmarks for 3D human pose estimation, namely Human3.6M, MPI-INF-3DHP and 3DPW.

Figures (4)

• Figure 1: The overview of FMPose. From a set of heatmaps predicted by the 2D pose detector HRNet, top-$k$ arguments are used as the input $z$ for the GCN. The GCN then computes the lifting condition vector $c$ by aggregating 2D information via the learned adjacency matrix $A$. Given the condition, the continuous flow model iteratively moves the initial pose $x_0$, drawn from the Gaussian distribution, towards the plausible 3D pose $x_1$ using a numerical ODE solver. The movement direction, velocity $\Dot{x}_t$, is estimated by the neural network $f_{\theta}$ trained with the OT path, given the current state $(x_t,t)$ and the condition $c$ as inputs. Each initial $x_0$ produces a corresponding 3D pose $x_1$, thus we can generate a set of hypotheses by sampling $x_0 \sim \mathcal{N}(0,I)$.
• Figure 2: Qualitative results of FMPose on the Human3.6M, MPI-INF-3DHP and 3DPW datasets. The human pose contains 17 joints, with the right side encoded with blue, left side with orange, and the middle with green. All 200 hypotheses are drawn for each example, with the best hypothesis in bold. The examples are randomly selected to cover all scenarios: indoor, complex background and outdoor. It is visually clear that FMPose performs well on different setups. On challenging scenarios, i.e. fully occluded joints from the camera view, FMPose produces a wider range of hypotheses along the optical axis, providing the additional uncertainty awareness for downstream tasks.
• Figure 3: Additional qualitative results on in-the-wild human poses. First two rows are examples taken from the Leeds Sport Pose dataset Johnson11, consisting of complex sport poses in a wide variety of background environments. The next two rows are examples from the KTH Football dataset Kazemi13, which mostly contains footage of football players playing on the field, with green grass. All 200 hypotheses are visualized, with the mean pose is bold. In both cases, FMPose performs well with plausible 3D reconstructions.
• Figure 4: Visualization of the adjacency matrix learned from the Human3.6M dataset (left) and the 3DPW dataset (right). The brighter the entry, the more correlation between the two joints towards the 3D pose generations of FMPose.