Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On

TL;DR

The paper addresses the challenge of realistic image-based virtual try-on by preserving garment texture and body details without garment warping or extra encoders. It introduces Texture-Preserving Diffusion (TPD), which uses Self-Attention-based Texture Transfer by spatially concatenating the garment and masked person images and leveraging self-attention in the diffusion UNet, along with a Decoupled Mask Prediction to generate accurate inpainting masks. The key contributions are a warping-free texture transfer mechanism, an automatic mask predictor that adapts to each garment, and state-of-the-art results on VITON and VITON-HD. This approach enables efficient and faithful virtual try-on for garment-to-person and person-to-person tasks and extends diffusion-based image editing without reliance on specialized encoders, with potential for broader applications and future work on more complex backgrounds.

Abstract

Image-based virtual try-on is an increasingly important task for online shopping. It aims to synthesize images of a specific person wearing a specified garment. Diffusion model-based approaches have recently become popular, as they are excellent at image synthesis tasks. However, these approaches usually employ additional image encoders and rely on the cross-attention mechanism for texture transfer from the garment to the person image, which affects the try-on's efficiency and fidelity. To address these issues, we propose an Texture-Preserving Diffusion (TPD) model for virtual try-on, which enhances the fidelity of the results and introduces no additional image encoders. Accordingly, we make contributions from two aspects. First, we propose to concatenate the masked person and reference garment images along the spatial dimension and utilize the resulting image as the input for the diffusion model's denoising UNet. This enables the original self-attention layers contained in the diffusion model to achieve efficient and accurate texture transfer. Second, we propose a novel diffusion-based method that predicts a precise inpainting mask based on the person and reference garment images, further enhancing the reliability of the try-on results. In addition, we integrate mask prediction and image synthesis into a single compact model. The experimental results show that our approach can be applied to various try-on tasks, e.g., garment-to-person and person-to-person try-ons, and significantly outperforms state-of-the-art methods on popular VITON, VITON-HD databases.

Figures (8)

• Figure 1: The sample try-on images synthesized by our Texture-Preserving Diffusion (TPD) model. In each triplet, the two left images are the original person and garment images from VITON-HD choi2021viton database. The right one depicts the synthesized image.
• Figure 2: Comparisons between different virtual try-on mechanisms. (a) The warping-based mechanism. (b) The cross-attention-based warping-free mechanism. (c) Our self-attention-based mechanism. $A$ represents the attention weight of a specific query-key pair.
• Figure 3: An overview of our framework. (a) In the training phase, we begin with the original person image $S$ and a randomly augmented mask $c_m$. $c_m$ is obtained by interpolating between the original clothing area $M_s$ and the bounding box $b_s$. The augmented mask $c_m$, the masked person image $c_m \odot S$, the pose map $c_p$, and the dense pose $c_d$ serve as the auxiliary input for the denoising UNet. Furthermore, the reference garment $C$ is concatenated with each of the auxiliary input along the spatial dimension as the context of the self-attention mechanism. (b) The inference phase is divided into two stages. In the first stage, we predict the clothing area $m_0^{s1}$ for the new garment $C^*$ on the person. We obtain $c_m^{s2}$ via element-wise multiplication between $m_0^{s1}$ and $M_s$. In the second stage, $c_m^{s2}$ is utilized as an accurate inpainting mask, enabling the diffusion model to produce high-fidelity try-on images. For clarity, we omit the predicted concatenated garments from the results of both stages.
• Figure 4: The qualitative comparisons between our method and state-of-the-art methods on VITON-HD choi2021viton database.
• Figure 5: The qualitative comparisons between our method and state-of-the-art methods on VITON han2018viton database.