Pose Priors from Language Models

TL;DR

ProsePose harnesses language-based priors from large multimodal models to guide 3D human pose estimation in scenes with self- or inter-person contact. It converts LMM-derived contact descriptions into differentiable losses, aggregates across multiple samples to mitigate hallucination, and jointly optimizes SMPL-X parameters with standard priors and 2D keypoint cues. The approach improves cross-dataset pose accuracy (PA-MPJPE) and contact correctness (PCC) for two-person interactions and self-contact yoga poses without requiring explicit contact annotations for training. This language-guided framework offers a scalable path to richer pose priors and can catalyze the creation of larger contact datasets, with broader implications for contact-aware 3D reasoning in vision tasks.

Abstract

Language is often used to describe physical interaction, yet most 3D human pose estimation methods overlook this rich source of information. We bridge this gap by leveraging large multimodal models (LMMs) as priors for reconstructing contact poses, offering a scalable alternative to traditional methods that rely on human annotations or motion capture data. Our approach extracts contact-relevant descriptors from an LMM and translates them into tractable losses to constrain 3D human pose optimization. Despite its simplicity, our method produces compelling reconstructions for both two-person interactions and self-contact scenarios, accurately capturing the semantics of physical and social interactions. Our results demonstrate that LMMs can serve as powerful tools for contact prediction and pose estimation, offering an alternative to costly manual human annotations or motion capture data. Our code is publicly available at https://prosepose.github.io.

Figures (28)

• Figure 1: Optimizing human-to-human contacts in 3D pose. Our approach leverages the semantic priors of a Large Multimodal Model (LMM) to infer meaningful information about physical contact from images. Instead of relying on human annotations or motion capture data, we extract not only descriptive insights ("... engaged in a dance or embrace ...") but also structured constraints between body parts (underlined). By incorporating these LMM-derived constraints, we refine initial 3D human pose estimates, achieving realistic and semantically consistent reconstructions of contact. This scalable approach opens up new possibilities for contact-aware pose estimation without explicit contact annotations, making it a promising alternative to traditional methods.
• Figure 2: LMM-guided Pose Estimation.(a) Method overview: ProsePose takes as input an image of one or two people in contact. We first obtain initial pose estimates for each person from a pose regressor. Then we use an LMM to generate contact constraints, each of which is a pair of body parts that should be touching. This list of contacts is converted into a loss function $\mathcal{L}_{\text{LMM}}$. We optimize the pose estimates using $\mathcal{L}_{\text{LMM}}$ and other losses to produce a refined estimate of each person's pose that respects the predicted contacts. (b) Defining contact constraints: Given an image $\boldsymbol{I}$, we can lift each individual into corresponding 3D meshes $\boldsymbol{V}$. A contact constraint $\boldsymbol{c}$ is a pair of regions $(\boldsymbol{R}_a, \boldsymbol{R}_b$) in contact. The loss is defined in terms of the distance between the vertices $(\boldsymbol{v}_a, \boldsymbol{v}_b$) on the mesh.
• Figure 3: Two-person examples We show qualitative results from ProsePose , BUDDI muller2023generative, and the contact heuristic. For each example, we show GPT4-V's top 3 constraints and the number of times each constraint was predicted across all 20 samples. Our method correctly reconstructs people in a variety of interactions, and the predicted constraints generally align with each interaction type.
• Figure 4: Single-person examples We show qualitative results from ProsePose , HMR2 goel2023humans, and HMR2-optim on complex yoga poses. Each example also shows the constraints that are predicted by the LMM at least $f=10$ times (and are thus used to compute $\mathcal{L}_{\text{LMM}}$) with their counts. ProsePose correctly identifies self-contact points and optimizes the poses to respect these contacts.
• Figure 5: More samples improve pose estimation. On the FlickrCI3D validation set, taking more samples from the LMM and averaging the resulting loss functions improves joint PA-MPJPE (left) and average PCC (right).