Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis

TL;DR

The paper tackles overfitting in pose-guided person image synthesis by removing reliance on textual prompts and learning semantic source-image understanding directly from images. It introduces CFLD, a diffusion-based framework with a perception-refined decoder for coarse-grained prompts and a hybrid-granularity attention module to inject multi-scale fine-grained texture as bias, all while decoupling pose and appearance controls via a lightweight pose adapter. With image-only training and classifier-free guidance, CFLD achieves state-of-the-art results on DeepFashion across quantitative metrics and user studies, and supports appearance editing and texture interpolation without extra supervision. This approach broadens diffusion-based PGPIS to settings lacking image-caption data and demonstrates strong generalization to unseen poses and clothing textures.

Abstract

Diffusion model is a promising approach to image generation and has been employed for Pose-Guided Person Image Synthesis (PGPIS) with competitive performance. While existing methods simply align the person appearance to the target pose, they are prone to overfitting due to the lack of a high-level semantic understanding on the source person image. In this paper, we propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for PGPIS. In the absence of image-caption pairs and textual prompts, we develop a novel training paradigm purely based on images to control the generation process of a pre-trained text-to-image diffusion model. A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt. This allows for the decoupling of fine-grained appearance and pose information controls at different stages, and thus circumventing the potential overfitting problem. To generate more realistic texture details, a hybrid-granularity attention module is proposed to encode multi-scale fine-grained appearance features as bias terms to augment the coarse-grained prompt. Both quantitative and qualitative experimental results on the DeepFashion benchmark demonstrate the superiority of our method over the state of the arts for PGPIS. Code is available at https://github.com/YanzuoLu/CFLD.

Figures (10)

• Figure 1: (a) The appearance of person images varies significantly given only a textual prompt for image generation by using Stable Diffusion stable_diffusion or ControlNet controlnet with OpenPose guidance openpose. (b) Simply aligning the source appearance to the target pose without a semantic understanding of person image can easily lead to overfitting, such that the generated images become distorted and unnatural. (c) Our method learns the coarse-grained prompt for a comprehensive perception of the source image and injects fine-grained appearance features as bias terms, thus generating high-quality images with better generalization performance.
• Figure 2: (a) Architecture of our proposed Coarse-to-Fine Latent Diffusion (CFLD) method. For pose-guided latent diffusion, we incorporate a lightweight pose adapter $\mathcal{H}_P$ from t2i_adapter to add its output feature maps to the end of each down-sampling block of the pre-trained UNet $\mathcal{H}_N$ for efficient structural guidance. To achieve a coarse-to-fine appearance control, we propose a perception-refined decoder $\mathcal{H}_D$ and hybrid-granularity attention module $\mathcal{H}_A$, both of which take the multi-scale feature maps from a source image encoder $\mathcal{H}_S$ as inputs. (b) The coarse-grained prompt is obtained by refining the learnable queries progressively in our proposed $\mathcal{H}_D$. (c) We encode the multi-scale fine-grained appearance features as bias terms in the up-sampling blocks for better texture details within $\mathcal{H}_A$.
• Figure 3: Qualitative comparisons with state-of-the-arts. To clarify the relation between objective and subjective metrics, we demonstrate the LPIPS measures and label the images with the first and second highest votes from user opinions as red and blue respectively.
• Figure 4: User study results in terms of R2G, G2R and Jab metrics.
• Figure 5: Qualitative ablation results. Our approach has a high-level understanding of the source image rather than forced alignment. It is also less prone to overfitting through the complementary coarse-grained prompts and fine-grained appearance biasing.