Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on

Chenhui Wang, Tao Chen, Zhihao Chen, Zhizhong Huang, Taoran Jiang, Qi Wang, Hongming Shan

TL;DR

FLDM-VTON introduces a faithful latent diffusion framework for virtual try-on that tackles unfaithfulness in clothing details by (1) using warped clothes as both the diffusion starting point and a local condition, (2) employing a clothes-flattening network for clothes-consistent supervision, and (3) implementing clothes-posterior sampling for faithful inference. The method integrates a global DINO-V2 controller and a dedicated training objective that combines standard diffusion loss with a clothes-consistency loss, yielding improved retention of texture, pattern, and text. Experimental results on VITON-HD and Dress Code show FLDM-VTON outperforms state-of-the-art baselines in fidelity while maintaining competitive realism, and qualitative results extend to real-world images. The work offers practical impact for high-resolution VTON tasks, enabling more trustworthy clothing detail transfer, with broader implications for e-commerce and digital fashion visualization.

Abstract

Despite their impressive generative performance, latent diffusion model-based virtual try-on (VTON) methods lack faithfulness to crucial details of the clothes, such as style, pattern, and text. To alleviate these issues caused by the diffusion stochastic nature and latent supervision, we propose a novel Faithful Latent Diffusion Model for VTON, termed FLDM-VTON. FLDM-VTON improves the conventional latent diffusion process in three major aspects. First, we propose incorporating warped clothes as both the starting point and local condition, supplying the model with faithful clothes priors. Second, we introduce a novel clothes flattening network to constrain generated try-on images, providing clothes-consistent faithful supervision. Third, we devise a clothes-posterior sampling for faithful inference, further enhancing the model performance over conventional clothes-agnostic Gaussian sampling. Extensive experimental results on the benchmark VITON-HD and Dress Code datasets demonstrate that our FLDM-VTON outperforms state-of-the-art baselines and is able to generate photo-realistic try-on images with faithful clothing details.

FLDM-VTON: Faithful Latent Diffusion Model for Virtual Try-on

TL;DR

FLDM-VTON introduces a faithful latent diffusion framework for virtual try-on that tackles unfaithfulness in clothing details by (1) using warped clothes as both the diffusion starting point and a local condition, (2) employing a clothes-flattening network for clothes-consistent supervision, and (3) implementing clothes-posterior sampling for faithful inference. The method integrates a global DINO-V2 controller and a dedicated training objective that combines standard diffusion loss with a clothes-consistency loss, yielding improved retention of texture, pattern, and text. Experimental results on VITON-HD and Dress Code show FLDM-VTON outperforms state-of-the-art baselines in fidelity while maintaining competitive realism, and qualitative results extend to real-world images. The work offers practical impact for high-resolution VTON tasks, enabling more trustworthy clothing detail transfer, with broader implications for e-commerce and digital fashion visualization.

Abstract

Despite their impressive generative performance, latent diffusion model-based virtual try-on (VTON) methods lack faithfulness to crucial details of the clothes, such as style, pattern, and text. To alleviate these issues caused by the diffusion stochastic nature and latent supervision, we propose a novel Faithful Latent Diffusion Model for VTON, termed FLDM-VTON. FLDM-VTON improves the conventional latent diffusion process in three major aspects. First, we propose incorporating warped clothes as both the starting point and local condition, supplying the model with faithful clothes priors. Second, we introduce a novel clothes flattening network to constrain generated try-on images, providing clothes-consistent faithful supervision. Third, we devise a clothes-posterior sampling for faithful inference, further enhancing the model performance over conventional clothes-agnostic Gaussian sampling. Extensive experimental results on the benchmark VITON-HD and Dress Code datasets demonstrate that our FLDM-VTON outperforms state-of-the-art baselines and is able to generate photo-realistic try-on images with faithful clothing details.
Paper Structure (41 sections, 7 equations, 16 figures, 4 tables)

This paper contains 41 sections, 7 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Image-based virtual try-on.
  5. Diffusion models.
  6. Methodology
  7. Try-on Diffusion with Clothes Priors
  8. Forward process.
  9. Reverse process.
  10. Clothes-consistent Faithful Supervision
  11. Clothes flattening network.
  12. Clothes-consistent supervision.
  13. Overview of Our FLDM-VTON
  14. Overview.
  15. Faithful inference.
  16. ...and 26 more sections

Figures (16)

  • Figure 1: Comparison between state-of-the-art baselines and our FLDM-VTON on the VITON-HD dataset.
  • Figure 2: Overview of the proposed FLDM-VTON. Our FLDM-VTON is trained with main and prior denoising input, constrained by DINO-V2, and the proposed clothes flattening network to generate the try-on images $\widehat{\boldsymbol{T}}$.
  • Figure 3: Illustration of the proposed clothes flattening network.
  • Figure 4: Qualitative results of different methods and ours on the VITON-HD dataset. Best viewed when zoomed in.
  • Figure 5: Qualitative results of different methods and ours on the Dress Code dataset. Best viewed when zoomed in.
  • ...and 11 more figures