Smart Fitting Room: A One-stop Framework for Matching-aware Virtual Try-on

TL;DR

The paper tackles the gap in virtual try-on systems by introducing matching-aware virtual try-on, which integrates fashion mix-and-match with try-on to ensure items are well-suited to a given person. It presents the Hybrid Matching-aware Virtual Try-On Framework (HMaVTON), combining a Retrieval-based Matching Module and a Generative Matching Module with an Adaptive Fusion Module to produce grounded, diverse outfit recommendations. The Virtual Try-on Module then warps and refines the selected garment using an appearance-flow guided generator and a diffusion-based denoiser, delivering high-quality try-on visuals. Evaluations on real-world datasets, including expert fashion designer studies, show that HMaVTON improves matching rationality/diversity and visual realism, offering a practical one-stop shopping experience.

Abstract

The development of virtual try-on has revolutionized online shopping by allowing customers to visualize themselves in various fashion items, thus extending the in-store try-on experience to the cyber space. Although virtual try-on has attracted considerable research initiatives, existing systems only focus on the quality of image generation, overlooking whether the fashion item is a good match to the given person and clothes. Recognizing this gap, we propose to design a one-stop Smart Fitting Room, with the novel formulation of matching-aware virtual try-on. Following this formulation, we design a Hybrid Matching-aware Virtual Try-On Framework (HMaVTON), which combines retrieval-based and generative methods to foster a more personalized virtual try-on experience. This framework integrates a hybrid mix-and-match module and an enhanced virtual try-on module. The former can recommend fashion items available on the platform to boost sales and generate clothes that meets the diverse tastes of consumers. The latter provides high-quality try-on effects, delivering a one-stop shopping service. To validate the effectiveness of our approach, we enlist the expertise of fashion designers for a professional evaluation, assessing the rationality and diversity of the clothes combinations and conducting an evaluation matrix analysis. Our method significantly enhances the practicality of virtual try-on. The code is available at https://github.com/Yzcreator/HMaVTON.

Figures (12)

• Figure 1: Mix-and-match and try-on are two essential fashion needs in daily life. We propose a one-stop system of Smart Fitting Room, which will generate or retrieve an apparel to match with the query and put it on the query image.
• Figure 2: A schematic of our one-stop framework. We adaptively fuse retrieval-based and generative matching results using the Adaptive Fusion Module, and provide high-quality try-on effects through the Virtual Try-on Module.
• Figure 3: In comparison with baseline, our approach produces a diverse range of clothes styles, such as long sleeves, short sleeves, and straps, with color matching that adheres to the principles of harmony, creating a soothing visual resonance in terms of brightness and saturation.
• Figure 4: Compared to other try-on models, our method exhibits superior performance in terms of clothes integrity restoration.
• Figure 5: We balance the retrieval and generation of outfits by controlling the threshold to regulate the proportion of retrieved clothes in all the recommended clothes.