Adapting Human Mesh Recovery with Vision-Language Feedback

TL;DR

This work tackles monocular human mesh recovery by unifying regression-based initialization with diffusion-based optimization under vision-language guidance. It introduces a multi-modal framework that uses ChatPose-derived interactive part descriptions and a shared text-pose latent space via VQ-VAE to constrain 3D pose and improve 2D-3D alignment. The diffusion process is conditioned on both 2D keypoints and text similarity, enabling gradients from vision and language to steer pose refinement and reduce depth ambiguity. Experimental results on 3DPW and Human3.6M show competitive improvements in MPJPE and PA-MPJPE, validating the value of incorporating vision-language feedback for robust 3D human mesh recovery.

Abstract

Human mesh recovery can be approached using either regression-based or optimization-based methods. Regression models achieve high pose accuracy but struggle with model-to-image alignment due to the lack of explicit 2D-3D correspondences. In contrast, optimization-based methods align 3D models to 2D observations but are prone to local minima and depth ambiguity. In this work, we leverage large vision-language models (VLMs) to generate interactive body part descriptions, which serve as implicit constraints to enhance 3D perception and limit the optimization space. Specifically, we formulate monocular human mesh recovery as a distribution adaptation task by integrating both 2D observations and language descriptions. To bridge the gap between text and 3D pose signals, we first train a text encoder and a pose VQ-VAE, aligning texts to body poses in a shared latent space using contrastive learning. Subsequently, we employ a diffusion-based framework to refine the initial parameters guided by gradients derived from both 2D observations and text descriptions. Finally, the model can produce poses with accurate 3D perception and image consistency. Experimental results on multiple benchmarks validate its effectiveness. The code will be made publicly available.

Figures (8)

• Figure 1: (a) Regression-based methods struggle with model-image alignment for challenging poses. (b) Optimization-based methods are prone to overfitting noisy 2D inputs and suffer from severe depth ambiguity. (c) Our method leverages prior knowledge from large vision-language models to improve both 2D and 3D performance.
• Figure 2: Pose-Text Alignment. We first train a discrete pose codebook via VQ-VAE. To bridge the gap between text and 3D pose modalities, we then train a text encoder to align the texts to body poses in latent space with contrastive learning.
• Figure 3: Overview of our method. Given an image, a large vision-language model is first used to extract detailed interactive descriptions for the body parts. An initial prediction is then made, followed by the construction of a diffusion-based framework that refines this prediction using multi-modal feedback. At each time step, the gradients of 2D keypoints are computed, along with the similarity loss between text embeddings and the pose, while image features from the backbone are concatenated to form the condition $c$, which is then fed into the diffusion model to estimate the noise. The distribution is updated based on this guidance, ultimately yielding accurate body pose estimations.
• Figure 4: Qualitative results. From left to right: RGB image, ProHMR kolotouros2021probabilistic, HMR2.0 HMR2.0, and our method. Our approach ensures accurate 3D joint positions with minimal depth ambiguity while achieving robust front-facing alignment.
• Figure 5: Prompt Generation. We use GPT-4 GPT4 to automatically generate 10 prompts, which are then verified and used with ChatPose ChatPose to describe each part of the human pose.