Multi-scale Attention Guided Pose Transfer

TL;DR

This work tackles pose transfer for a person across novel poses using a GAN-based framework with a dense, multi-scale attention mechanism. The generator employs two encoders (image and pose heatmaps) and a single decoder, with attention links at every resolution level to preserve fine details during pose manipulation, and a PatchGAN discriminator complements a loss that combines $\mathcal{L}_1$, adversarial, and perceptual terms from a pre-trained network. On DeepFashion, the approach achieves state-of-the-art qualitative and quantitative results, including improved PCKh and LPIPS scores, and is further validated by a user study. Beyond pose transfer, the method demonstrates versatility across conditional semantic reconstruction, virtual try-on, and skeleton-guided style transfer, indicating strong generalization without architectural changes.

Abstract

Pose transfer refers to the probabilistic image generation of a person with a previously unseen novel pose from another image of that person having a different pose. Due to potential academic and commercial applications, this problem is extensively studied in recent years. Among the various approaches to the problem, attention guided progressive generation is shown to produce state-of-the-art results in most cases. In this paper, we present an improved network architecture for pose transfer by introducing attention links at every resolution level of the encoder and decoder. By utilizing such dense multi-scale attention guided approach, we are able to achieve significant improvement over the existing methods both visually and analytically. We conclude our findings with extensive qualitative and quantitative comparisons against several existing methods on the DeepFashion dataset.

Figures (13)

• Figure 1: General overview of pose transfer using the proposed method.
• Figure 2: Architecture of the proposed generator. The generator takes the condition image $I_A^j$ along with the channel-wise concatenated pose heatmaps $(H_A^j, H_B^j)$ as inputs and generates an estimate $\hat{I}_B^j$ of the target image $I_B^j$.
• Figure 3: Architecture of the PatchGAN discriminator. The discriminator takes two channel-wise concatenated images, either $(I_A^j, I_B^j)$ or $(I_A^j, \hat{I}_B^j)$, as input and estimates a label map where, each label corresponds to the binary class probability of an input patch.
• Figure 4: Qualitative comparison among different pose transfer methods. $I_A^j$ denotes the condition image, $I_B^j$ denotes the target image and subsequent columns show the generated images by $\text{PG}^2$ma2017pose, Deformable GANs siarohin2018deformable, VUNet esser2018variational, PATN zhu2019progressive and our method.
• Figure 5: Ablation study -- Qualitative comparison among model variants. $I_A^j$ denotes the condition image, $I_B^j$ denotes the target image and each subsequent column shows the generated images by respective model variant.