Neural Clothing Tryer: Customized Virtual Try-On via Semantic Enhancement and Controlling Diffusion Model

TL;DR

This work tackles Customized Virtual Try-On (Cu-VTON), where a user-specified garment must be realistically pasted onto a customizable model with adjustable appearance, posture, and attributes. It proposes Neural Clothing Tryer (NCT), a diffusion-model framework augmented with a Semantic Enhancement (SE) module that aligns garment semantics via visual-language features and a Semantic Controlling (SC) module that enables dual-branch conditioning for garment detail preservation and pose/attribute editing. A cross-pairing data augmentation strategy using a traditional fitting model mitigates one-to-one training biases, enabling better generalization to diverse outfits and bodies. Experimental results on the Dress Code dataset, along with ablations and user studies, show that NCT achieves superior garment fidelity, natural customized appearances, and flexible editing capabilities compared with representative baselines, signaling practical viability for interactive Cu-VTON applications.

Abstract

This work aims to address a novel Customized Virtual Try-ON (Cu-VTON) task, enabling the superimposition of a specified garment onto a model that can be customized in terms of appearance, posture, and additional attributes. Compared with traditional VTON task, it enables users to tailor digital avatars to their individual preferences, thereby enhancing the virtual fitting experience with greater flexibility and engagement. To address this task, we introduce a Neural Clothing Tryer (NCT) framework, which exploits the advanced diffusion models equipped with semantic enhancement and controlling modules to better preserve semantic characterization and textural details of the garment and meanwhile facilitating the flexible editing of the model's postures and appearances. Specifically, NCT introduces a semantic-enhanced module to take semantic descriptions of garments and utilizes a visual-language encoder to learn aligned features across modalities. The aligned features are served as condition input to the diffusion model to enhance the preservation of the garment's semantics. Then, a semantic controlling module is designed to take the garment image, tailored posture image, and semantic description as input to maintain garment details while simultaneously editing model postures, expressions, and various attributes. Extensive experiments on the open available benchmark demonstrate the superior performance of the proposed NCT framework.

Figures (10)

• Figure 1: Illustration of the traditional and customized VTON tasks. Customized VTON devotes to superimpose a specified garment onto a model that can be customized in terms of appearance, posture, and additional attributes.
• Figure 2: An overall illustration of the NCT framework. We equip the diffusion model with the SE and SC modules, in which SE learns aligned features of both semantic description and garment images as an enhanced condition to the diffusion models, while the SC takes garment images, posture information, and semantic description of other attributes as input to preserve the garment details and simultaneously edits posture and other attributes.
• Figure 3: Comparison between model images in the Dress Code dataset and generation results obtained by NCT using the original Dress Code dataset for training.
• Figure 4: Visualization results of NCT and the competing methods on the Dress Code dataset. We present the results for both traditional VTON task with fixed posture and Cu-VTON task with various postures.
• Figure 5: Visual comparison results of NCT and competitive methods on the Dress Code dataset. We present the generation results of NCT under various attribute editing conditions.