StableVITON: Learning Semantic Correspondence with Latent Diffusion Model for Virtual Try-On

TL;DR

This work tackles image-based virtual try-on by enabling a pre-trained diffusion model to directly learn clothing–body semantic correspondence in latent space. It introduces StableVITON, which uses zero cross-attention within a latent-space conditioning framework, a clothing latent encoder, augmentation, and an attention total variation loss to preserve clothing details while exploiting diffusion priors. The method finetunes the diffusion model end-to-end without a separate warping network, achieving state-of-the-art results on in-domain and cross-domain tasks across VITON-HD, DressCode, and SHHQ-1.0, with strong qualitative and quantitative gains. User studies corroborate the improvements, highlighting practical potential for robust virtual try-on in real-world settings.

Abstract

Given a clothing image and a person image, an image-based virtual try-on aims to generate a customized image that appears natural and accurately reflects the characteristics of the clothing image. In this work, we aim to expand the applicability of the pre-trained diffusion model so that it can be utilized independently for the virtual try-on task.The main challenge is to preserve the clothing details while effectively utilizing the robust generative capability of the pre-trained model. In order to tackle these issues, we propose StableVITON, learning the semantic correspondence between the clothing and the human body within the latent space of the pre-trained diffusion model in an end-to-end manner. Our proposed zero cross-attention blocks not only preserve the clothing details by learning the semantic correspondence but also generate high-fidelity images by utilizing the inherent knowledge of the pre-trained model in the warping process. Through our proposed novel attention total variation loss and applying augmentation, we achieve the sharp attention map, resulting in a more precise representation of clothing details. StableVITON outperforms the baselines in qualitative and quantitative evaluation, showing promising quality in arbitrary person images. Our code is available at https://github.com/rlawjdghek/StableVITON.

Figures (16)

• Figure 1: Generated results of StableVITON: VITON-HD (the first row), SHHQ-1.0 (the first two images in the second row), and web-crawled images (the last two images in the second row). All results are generated using StableVITON trained on VITON-HD dataset.
• Figure 2: Visualization of the semantic correspondence learned by our StableVITON. We overlay the attention map for the clothing regions onto the generated images for visualization.
• Figure 3: For the virtual try-on task, StableVITON additionally takes three conditions: agnostic map, agnostic mask, and dense pose, as the input of the pre-trained U-Net, which serves as the query (Q) for the cross-attention. The feature map of the clothing is used as the key (K) and value (V) for the cross-attention and is conditioned on the UNet, as depicted in (b).
• Figure 4: Visualization of attention map from a zero cross-attention block of $32\time24$ resolution.
• Figure 5: Qualitative comparison with baselines in a single dataset setting (VITON-HD / VITON-HD). Best viewed when zoomed in.