Limb-Aware Virtual Try-On Network with Progressive Clothing Warping

TL;DR

PL-VTON tackles the challenge of image-based virtual try-on by introducing limb-aware, progressive clothing warping and limb-detail generation. It decomposes the problem into three modules—Progressive Clothing Warping (PCW) for fine-grained garment alignment, Person Parsing Estimator (PPE) for semantically guided body structure, and Limb-aware Texture Fusion (LTF) for coarse-to-fine texture synthesis with limb-aware guidance. Key contributions include a gravity-aware loss for realistic garment edges, a non-limb target parsing prior to target parsing, and an improved clothing-agnostic person representation to preserve non-clothing details. Empirical results on VITON and additional datasets show improvements in FID, SSIM, and PSNR, along with strong qualitative performance and user preferences, indicating practical impact for high-fidelity virtual try-on in e-commerce.

Abstract

Image-based virtual try-on aims to transfer an in-shop clothing image to a person image. Most existing methods adopt a single global deformation to perform clothing warping directly, which lacks fine-grained modeling of in-shop clothing and leads to distorted clothing appearance. In addition, existing methods usually fail to generate limb details well because they are limited by the used clothing-agnostic person representation without referring to the limb textures of the person image. To address these problems, we propose Limb-aware Virtual Try-on Network named PL-VTON, which performs fine-grained clothing warping progressively and generates high-quality try-on results with realistic limb details. Specifically, we present Progressive Clothing Warping (PCW) that explicitly models the location and size of in-shop clothing and utilizes a two-stage alignment strategy to progressively align the in-shop clothing with the human body. Moreover, a novel gravity-aware loss that considers the fit of the person wearing clothing is adopted to better handle the clothing edges. Then, we design Person Parsing Estimator (PPE) with a non-limb target parsing map to semantically divide the person into various regions, which provides structural constraints on the human body and therefore alleviates texture bleeding between clothing and body regions. Finally, we introduce Limb-aware Texture Fusion (LTF) that focuses on generating realistic details in limb regions, where a coarse try-on result is first generated by fusing the warped clothing image with the person image, then limb textures are further fused with the coarse result under limb-aware guidance to refine limb details. Extensive experiments demonstrate that our PL-VTON outperforms the state-of-the-art methods both qualitatively and quantitatively.

Figures (22)

• Figure 1: Fine-grained Clothing warping and realistic limb detail generation are challenging in image-based virtual try-on, especially when transforming between long-sleeved and short-sleeved clothing. Compared with existing methods, our proposed method achieves superior performance in these cases.
• Figure 2: The finesse and stability of different clothing warping strategies, where TPS represents the warped results based on thin-plate spline transformation, AF represents the warped results based on an appearance flow, and the dotted lines are drawn to show the inconsistency in location and size of the person's original clothing and the in-shop clothing more intuitively.
• Figure 3: The overview of the proposed PL-VTON. (a) PCW takes an in-shop clothing image $C$, an in-shop clothing mask $M$, a source parsing map $P^s$, and a keypoint map $K$ as inputs to generate a warped clothing image $C_w$ through a two-stage alignment strategy. (b) PPE first synthesizes a non-limb target parsing map $P^t_{nl}$ through a semantic combination and then uses this prior parsing result to predict a target parsing map $P^t$. (c) LTF performs a texture fusion process between the warped clothing and the person image based on a set of limb map patches $L_p$ and outputs the try-on result $I_f$.
• Figure 4: The illustration of Progressive Clothing Warping (PCW). Given an in-shop clothing image $C$, an in-shop clothing mask $M$, an occluded source parsing map $P^s_{occ}$, and a keypoint map $K$, PCW first estimates the parameters of translation and scaling through a two-branch network, which are used to adjust the location and size of the clothing in the in-shop clothing image to get a pre-aligned clothing image $C_a$. Then a set of sub-flows are predicted and aggregated to one through the multi-scale flow predictor, which is applied to $C_a$ and $M_a$ to get a warped clothing image $C_w$ and its mask $M_w$.
• Figure 5: The human shape map contains geometric information of the person's original clothing such as the collar shape, even if it has been down-sampled.