Direct Reward Fine-Tuning on Poses for Single Image to 3D Human in the Wild

TL;DR

DrPose, Direct Reward fine-tuning algorithm on Poses, which enables post-training of a multi-view diffusion model on diverse poses without requiring expensive 3D human assets to address limitation of available 3D human datasets with diverse poses.

Abstract

Single-view 3D human reconstruction has achieved remarkable progress through the adoption of multi-view diffusion models, yet the recovered 3D humans often exhibit unnatural poses. This phenomenon becomes pronounced when reconstructing 3D humans with dynamic or challenging poses, which we attribute to the limited scale of available 3D human datasets with diverse poses. To address this limitation, we introduce DrPose, Direct Reward fine-tuning algorithm on Poses, which enables post-training of a multi-view diffusion model on diverse poses without requiring expensive 3D human assets. DrPose trains a model using only human poses paired with single-view images, employing a direct reward fine-tuning to maximize PoseScore, which is our proposed differentiable reward that quantifies consistency between a generated multi-view latent image and a ground-truth human pose. This optimization is conducted on DrPose15K, a novel dataset that was constructed from an existing human motion dataset and a pose-conditioned video generative model. Constructed from abundant human pose sequence data, DrPose15K exhibits a broader pose distribution compared to existing 3D human datasets. We validate our approach through evaluation on conventional benchmark datasets, in-the-wild images, and a newly constructed benchmark, with a particular focus on assessing performance on challenging human poses. Our results demonstrate consistent qualitative and quantitative improvements across all benchmarks. Project page: https://seunguk-do.github.io/drpose.

Figures (16)

• Figure 1: We propose DrPose (Direct Reward Fine-tuning on Poses), a method to post-train a multi-view diffusion model for enhanced posture of reconstructed 3D humans in dynamic and acrobatic scenarios.
• Figure 2: Illustration of DrPose presented in Algo \ref{['alg:alg']}. A Denoising multi-view U-Net $\epsilon_{\omega}$ is trained to minimize $\mathcal{L}_{\textrm{total}}=\mathcal{L}_{\textrm{reward}}+w_{\mathrm{KL}}\cdot \mathcal{L}_{\mathrm{KL}}$. Multi-view latent images $x_0$ are generated from $x_T \sim \mathcal{N}(0, \mathbf{I})$, and $\mathcal{L}_{\textrm{reward}}$ is computed from $x_0$ and the ground-truth 3D human pose $\theta$. For efficiency, only a subset of denoising steps is sampled for training. Concurrently, $\mathcal{L}_{\mathrm{KL}}$ is computed as the KL divergence between $\epsilon_{\omega}$ and the frozen reference U-Net $\epsilon_{w_0}$ at intermediate denoising steps. For clarity, only 3 of the 6 multi-view images are shown.
• Figure 3: Construction process for DrPose15K. We employ a pose-conditioned video generator model men2025mimo to generate single-view images from 3D human poses.
• Figure 4: Comparison of pose diversity between conventional 3D human datasets yu2021function4dho2023learning and our proposed DrPose15K. Our dataset has a higher standard deviation of SMPL-X joint locations than other datasets.
• Figure 5: Overview of our 3D human reconstruction pipeline. In this pipeline, the multi-view normal and RGB images are generated from the input image using an image-to-multi-view (I2MV) diffusion model. These images are then converted into a 3D representation using explicit human carving pshuman_li_2024. In this work, we propose post-training the I2MV diffusion model to achieve better alignment with accurate poses in dynamic and acrobatic scenarios. For clarity, only 3 of the 6 multi-view images are displayed for normal maps and RGB images.