BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training

TL;DR

BooW-VTON is introduced, the mask-free virtual try-on diffusion model, which delivers SOTA try-on quality without parsing cost and ensures the model’s mask-free try-on capability by creating high-quality pseudo-data and enhances its handling of complex spatial information through effective in-the-wild data augmentation.

Abstract

Image-based virtual try-on is an increasingly popular and important task to generate realistic try-on images of the specific person. Recent methods model virtual try-on as image mask-inpaint task, which requires masking the person image and results in significant loss of spatial information. Especially, for in-the-wild try-on scenarios with complex poses and occlusions, mask-based methods often introduce noticeable artifacts. Our research found that a mask-free approach can fully leverage spatial and lighting information from the original person image, enabling high-quality virtual try-on. Consequently, we propose a novel training paradigm for a mask-free try-on diffusion model. We ensure the model's mask-free try-on capability by creating high-quality pseudo-data and further enhance its handling of complex spatial information through effective in-the-wild data augmentation. Besides, a try-on localization loss is designed to concentrate on try-on area while suppressing garment features in non-try-on areas, ensuring precise rendering of garments and preservation of fore/back-ground. In the end, we introduce BooW-VTON, the mask-free virtual try-on diffusion model, which delivers SOTA try-on quality without parsing cost. Extensive qualitative and quantitative experiments have demonstrated superior performance in wild scenarios with such a low-demand input.

Figures (24)

• Figure 1: Boow-VTON achieves realistic virtual try-on results for in-the-wild scenarios, with complex foreground and background elements. While performing natural try-ons for garments, we faithfully preserve the details of the person (such as accessories, muscles, and skin) and the style of the image (including lighting, shadows, and artistic style).
• Figure 2: Top: Shop try-on, accurate masks can be obtained through precise human parsing results. The masked person has destroyed spatial information (represented by depth maps). Bottom: In-the-wild try-on, the accuracy of the mask region decreases as the interference of complex scenarios. The masked person suffers from severe loss of spatial information.
• Figure 3: Overview of high-quality in-the-wild try-on model training pipeline. (a) Data prepare for pseudo training pairs. Use the mask-based model with a refined inference pipeline to generate high-quality pseudo-person images. (b-i) Implementation of in-the-wild pseudo pairs. Add background and foreground to the person's image using image stacking. (b-ii) Training process of mask-free try-on diffusion model. Try-on U-Net is trained to determine the try-on regions from the person image $P'$. Use $M^{Aug}$ to constrain the regions for garment alignment and replacement in the attention layer. $M^{Aug}$ is used only during training.
• Figure 4: Creating fore/back-ground by synthesizing data.
• Figure 5: Using try-on mask $M^{Aug}$ to guide the model’s attention to the correct try-on areas.