TED-VITON: Transformer-Empowered Diffusion Models for Virtual Try-On

TL;DR

TED-VITON tackles the limitations of text-to-image diffusion backbones for Virtual Try-On by migrating VTO to a Diffusion Transformer (DiT) framework and addressing garment detail and text fidelity. It introduces a Garment Semantic (GS) Adapter, a Text Preservation Loss, and an LLM-guided prompt constraint to align high-level garment semantics with detailed visual rendering, while leveraging DiT-GarmentNet and DiT-TryOnNet for joint garment-person synthesis. The model uses GPT-4o-generated garment descriptions to provide rich semantic conditioning and employs a prior-preservation-inspired loss to maintain text clarity during fine-tuning. Quantitative and qualitative evaluations on VITON-HD and DressCode demonstrate state-of-the-art results in LPIPS, CLIP-I, FID, and KID, with superior text rendering of logos and multi-line text, validating TED-VITON as a scalable, high-fidelity solution for realistic VTO in varied poses and lighting.

Abstract

Recent advancements in Virtual Try-On (VTO) have demonstrated exceptional efficacy in generating realistic images and preserving garment details, largely attributed to the robust generative capabilities of text-to-image (T2I) diffusion backbones. However, the T2I models that underpin these methods have become outdated, thereby limiting the potential for further improvement in VTO. Additionally, current methods face notable challenges in accurately rendering text on garments without distortion and preserving fine-grained details, such as textures and material fidelity. The emergence of Diffusion Transformer (DiT) based T2I models has showcased impressive performance and offers a promising opportunity for advancing VTO. Directly applying existing VTO techniques to transformer-based T2I models is ineffective due to substantial architectural differences, which hinder their ability to fully leverage the models' advanced capabilities for improved text generation. To address these challenges and unlock the full potential of DiT-based T2I models for VTO, we propose TED-VITON, a novel framework that integrates a Garment Semantic (GS) Adapter for enhancing garment-specific features, a Text Preservation Loss to ensure accurate and distortion-free text rendering, and a constraint mechanism to generate prompts by optimizing Large Language Model (LLM). These innovations enable state-of-the-art (SOTA) performance in visual quality and text fidelity, establishing a new benchmark for VTO task. Project page: https://zhenchenwan.github.io/TED-VITON/

Figures (6)

• Figure 1: We propose an implementation of Virtual Try-On using the Diffusion Transformer (DiT) architecture, which demonstrates state-of-the-art visual quality by preserving fine garment details and text clarity, even under challenging conditions involving complex human poses and diverse lighting environments.
• Figure 2: Overview of TED-VITON: We present the architecture of the proposed model along with details of its block modules. (a) Our model consists of 1) DiT-GarmentNet that encodes fine-grained features of $X_g$, 2) GS-Adapter ye_ip-adapter_2023 that captures higher-order semantics of garment image $X_g$, and 3) DiT-TryOnNet, the main Transformer for processing person images. The Transformer input is formed by concatenating the noised latents $X_t$ with the segmentation mask $m$, masked image $\mathcal{E}(X_\text{model})$, and Densepose guler_densepose_2018$\mathcal{E}(x_{\text{pose}})$. Additionally, a detailed description of the garment (e.g., “[D]: The clothing item is a black T-shirt...”) is generated through an LLM and fed as input to both the DiT-GarmentNet and DiT-TryOnNet. The model aims to preserve garment-specific details through a text preservation loss, which ensures that key textual features are retained. (b) Intermediate features from DiT-TryOnNet and DiT-GarmentNet are concatenated. These are then refined through joint-attention and cross-attention layers, with the GS-Adapter further contributing to the refinement process. In this architecture, the DiT-TryOnNet and GS-Adapter modules are fine-tuned, while other components remain frozen.
• Figure 3: Qualitative comparison with baseline methods. This figure demonstrates the superior performance of our model compared to various SOTA approaches. Zooming in reveals finer details.
• Figure 4: An ablation study on the key components of TED-VITON.
• Figure 5: User study results based on 10 selected pairs from the VITON-HD choi_viton-hd_2021 dataset: 5 pairs assessed text and logo preferences, and the other 5 focused on pattern and texture preferences.