FitDiT: Advancing the Authentic Garment Details for High-fidelity Virtual Try-on

TL;DR

A novel garment perception enhancement technique, termed FitDiT, designed for high-fidelity virtual try-on using Diffusion Transformers (DiT) allocating more parameters and attention to high-resolution features, and introduces frequency-domain learning by customizing a frequency distance loss to enhance high-frequency garment details.

Abstract

Although image-based virtual try-on has made considerable progress, emerging approaches still encounter challenges in producing high-fidelity and robust fitting images across diverse scenarios. These methods often struggle with issues such as texture-aware maintenance and size-aware fitting, which hinder their overall effectiveness. To address these limitations, we propose a novel garment perception enhancement technique, termed FitDiT, designed for high-fidelity virtual try-on using Diffusion Transformers (DiT) allocating more parameters and attention to high-resolution features. First, to further improve texture-aware maintenance, we introduce a garment texture extractor that incorporates garment priors evolution to fine-tune garment feature, facilitating to better capture rich details such as stripes, patterns, and text. Additionally, we introduce frequency-domain learning by customizing a frequency distance loss to enhance high-frequency garment details. To tackle the size-aware fitting issue, we employ a dilated-relaxed mask strategy that adapts to the correct length of garments, preventing the generation of garments that fill the entire mask area during cross-category try-on. Equipped with the above design, FitDiT surpasses all baselines in both qualitative and quantitative evaluations. It excels in producing well-fitting garments with photorealistic and intricate details, while also achieving competitive inference times of 4.57 seconds for a single 1024x768 image after DiT structure slimming, outperforming existing methods.

Figures (17)

• Figure 1: FitDiT demonstrates exceptional performance in virtual try-on, addressing challenges related to texture-aware preservation and size-aware fitting across various scenarios.
• Figure 2: FitDiT employs a two-stage training strategy. In the first stage, Garment Priors Evolution is utilized to fine-tune GarmentDiT for enhanced clothing feature extraction. In the second stage, we customize the DiT blocks through structure slimming, garment condition modulation, and high-resolution garment feature injection, resulting in DenoisingDiT for the try-on. DenoisingDiT is trained jointly using frequency loss and denoising loss.
• Figure 3: Previous works tend to fill the entire inpainting area due to a strict mask strategy. In contrast, FitDiT can accurately restore the shape of the garment with the dilated-relaxed mask strategy.
• Figure 4: Frequency domain gaps between the real and the generated images by different algorithms.
• Figure 5: Attention-related parameter ratios at various resolutions.