ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization

TL;DR

ADHMR tackles the ill-posed problem of single-image human mesh recovery by aligning diffusion-based HMR with image cues through direct preference optimization. It introduces HMR-Scorer to assess mesh predictions and generate a synthetic winner–loser preference dataset, which is used to finetune the base diffusion model via a Diffusion Direct Preference Optimization objective. The framework also leverages HMR-Scorer for automated data cleaning to improve other HMR models with less data. Empirical results show ADHMR achieves state-of-the-art performance and robustness in wild conditions, with substantial gains over ScoreHypo and benefits from data cleaning across baselines. Overall, ADHMR provides a scalable, alignment-focused approach for diffusion-based HMR with practical implications for real-world 3D human reconstruction.

Abstract

Human mesh recovery (HMR) from a single image is inherently ill-posed due to depth ambiguity and occlusions. Probabilistic methods have tried to solve this by generating numerous plausible 3D human mesh predictions, but they often exhibit misalignment with 2D image observations and weak robustness to in-the-wild images. To address these issues, we propose ADHMR, a framework that Aligns a Diffusion-based HMR model in a preference optimization manner. First, we train a human mesh prediction assessment model, HMR-Scorer, capable of evaluating predictions even for in-the-wild images without 3D annotations. We then use HMR-Scorer to create a preference dataset, where each input image has a pair of winner and loser mesh predictions. This dataset is used to finetune the base model using direct preference optimization. Moreover, HMR-Scorer also helps improve existing HMR models by data cleaning, even with fewer training samples. Extensive experiments show that ADHMR outperforms current state-of-the-art methods. Code is available at: https://github.com/shenwenhao01/ADHMR.

Figures (2)

• Figure 1: Overview of ADHMR. We aim to finetune a probabilistic HMR base model that generates multiple human mesh predictions conditioned on the input image. We first train the HMR-Scorer that assesses the reconstruction quality given an image and corresponding human mesh predictions. The reconstruction quality annotations $Q^*$ are computed using standard HMR metrics, including PVE $Q^{pve}$, MPJPE $Q^{mpjpe}$, PA-MPJPE $Q^{pajpe}$, and PA-PVE $Q^{papve}$. Next, we construct a synthetic human preference dataset, where each sample is a $\left<\text{winner}, \text{loser}\right>$ prediction pair rated by the HMR-Scorer. Finally, ADHMR uses this synthetic human preference dataset to finetune the base model to preferentially generate predictions that are more plausible and better aligned with the image cues.
• Figure 2: Qualitative comparison of the state-of-the-art probabilistic model ScoreHypo xu2024scorehypo and our ADHMR. Our framework significantly improves image alignment and in-the-wild robustness. (a) $\sim$ (f) are from the 3DPW 3dpw dataset, and (g) $\sim$ (h) are challenging in-the-wild images.