DiffProxy: Multi-View Human Mesh Recovery via Diffusion-Generated Dense Proxies

TL;DR

DiffProxy addresses the challenge of real-world human mesh recovery by leveraging diffusion priors trained on large-scale synthetic multi-view data to generate dense, pixel-to-surface proxies with multi-view consistency. The method combines a diffusion-based proxy generator with hand refinement and test-time uncertainty weighting, followed by uncertainty-weighted SMPL-X reprojection fitting, achieving strong zero-shot generalization across five real-world benchmarks. Key contributions include multi-view epipolar-consistent proxy generation, a hand refinement module for finger fidelity, and an uncertainty-aware test-time scaling strategy that improves robustness under occlusion and partial views. This approach demonstrates that diffusion-based priors can effectively bridge synthetic-to-real generalization for dense structured prediction tasks in 3D human reconstruction, with practical impact on scenarios where real annotations are scarce or biased.

Abstract

Human mesh recovery from multi-view images faces a fundamental challenge: real-world datasets contain imperfect ground-truth annotations that bias the models' training, while synthetic data with precise supervision suffers from domain gap. In this paper, we propose DiffProxy, a novel framework that generates multi-view consistent human proxies for mesh recovery. Central to DiffProxy is leveraging the diffusion-based generative priors to bridge the synthetic training and real-world generalization. Its key innovations include: (1) a multi-conditional mechanism for generating multi-view consistent, pixel-aligned human proxies; (2) a hand refinement module that incorporates flexible visual prompts to enhance local details; and (3) an uncertainty-aware test-time scaling method that increases robustness to challenging cases during optimization. These designs ensure that the mesh recovery process effectively benefits from the precise synthetic ground truth and generative advantages of the diffusion-based pipeline. Trained entirely on synthetic data, DiffProxy achieves state-of-the-art performance across five real-world benchmarks, demonstrating strong zero-shot generalization particularly on challenging scenarios with occlusions and partial views. Project page: https://wrk226.github.io/DiffProxy.html

Figures (7)

• Figure 1: DiffProxy is trained exclusively on synthetic data and achieves robust generalization to real-world scenarios. Our framework accepts diverse prompts (visual and textual), handles difficult poses, generalizes to challenging environments, and supports partial views with flexible view counts. Three key advantages: (i) Annotation bias-free—training on synthetic data avoids fitting biases from real datasets; (ii) Flexible—adapts to varying view counts, handles partial observations, and works across diverse capture conditions; (iii) Cross-data generalization—achieves strong performance across unseen real-world datasets without requiring real training pairs.
• Figure 2: Method overview. The figure illustrates our complete pipeline from multi-view images to final mesh recovery, which proceeds as follows: (a) given multi-view images and cameras parameters, the proxy generator produces per-view SMPL-X proxies $\mathbf{P}_v$; (b) hand-focused regions inferred from the body proxies are incorporated as additional views for hand refinement; (c) test-time scaling runs $K$ stochastic inference attempts, aggregates predictions through median (UV) and majority voting (segmentation), and computes pixel-wise uncertainty to produce a weight map $\mathbf{W}_v$ that guides fitting; (d) the body is fitted and then refined with hand-specific proxies to recover the final human mesh.
• Figure 3: Diffusion-based proxy generator architecture. Our model is built on Stable Diffusion 2.1 with a frozen UNet backbone, equipped with three conditioning signals ($\mathbf{c}_{\text{txt}}$, $\mathbf{c}_{\text{T2I}}$, $\mathbf{c}_{\text{DINO}}$) and four trainable attention modules ($\mathcal{A}_{\mathrm{text}}$, $\mathcal{A}_{\mathrm{img}}$, $\mathcal{A}_{\mathrm{cm}}$, $\mathcal{A}_{\mathrm{epi}}$) for multi-view consistent proxy generation.
• Figure 4: Qualitative comparison with baseline methods. Our method demonstrates: (i) bias-free predictions avoiding real-data annotation artifacts; (ii) strong generalization despite synthetic-only training; (iii) robustness to partial observations.
• Figure 5: Qualitative comparison of hand refinement. Hand refinement improves fitting quality and produces accurate finger details.