DiffBody: Diffusion-based Pose and Shape Editing of Human Images

TL;DR

This work tackles large pose and body-shape editing of human images while preserving the subject's identity. It introduces a one-shot pipeline that fits a textured SMPL-X model to the input, edits pose/shape, and then refines the result with a diffusion model in two stages: fullbody and facial refinement, guided by DreamBooth-style personalization and keypoint conditioning. Across multiple datasets, the method demonstrates superior pose-editing performance and more plausible body-shape edits, with ablations validating the contribution of iterative refinement, text-embedding optimization, and input reinitialization. The approach enables realistic, identity-preserving edits suitable for in-the-wild imagery, with potential for speed improvements and better handling of loose clothing in future work.

Abstract

Pose and body shape editing in a human image has received increasing attention. However, current methods often struggle with dataset biases and deteriorate realism and the person's identity when users make large edits. We propose a one-shot approach that enables large edits with identity preservation. To enable large edits, we fit a 3D body model, project the input image onto the 3D model, and change the body's pose and shape. Because this initial textured body model has artifacts due to occlusion and the inaccurate body shape, the rendered image undergoes a diffusion-based refinement, in which strong noise destroys body structure and identity whereas insufficient noise does not help. We thus propose an iterative refinement with weak noise, applied first for the whole body and then for the face. We further enhance the realism by fine-tuning text embeddings via self-supervised learning. Our quantitative and qualitative evaluations demonstrate that our method outperforms other existing methods across various datasets.

Figures (17)

• Figure 1: Results of editing the pose and body shape of a fullbody human image using our method. For various pose and body shape inputs, our method can generate realistic human images, while preserving the clothing textures and facial identity of reference person images.
• Figure 2: Limitations of the state-of-the-art methods for human pose editing by Ren et al. NTED and Bhunia et al. PIDM. These methods often struggle with identity preservation for input images not observed in a test set corresponding to a training set (see the red boxes).
• Figure 3: Problem of person image projection onto 3D parametric body models. As shown in the red boxes, the initial textured model often has artifacts when its pose and shape are edited.
• Figure 4: Refinement results of a textured model using a diffusion model with different noise strengths. Insufficient noise strength does not refine the visual artifacts, whereas excessively strong noise will significantly alter the person's identity.
• Figure 5: Overview of our method. Our method first computes a textured SMPL-X model from a reference person image. The SMPL-X model is then deformed according to given body pose, height, and weight parameters. To compensate for occluded and distorted textures resulting from texture projection, our method performs step-by-step refinement for the rendered image of the SMPL-X model using diffusion models.