IGR: Improving Diffusion Model for Garment Restoration from Person Image

TL;DR

IGR addresses the garment restoration task by conditioning a latent diffusion model on dual garment features: low-level $F_{ll}$ from IP-Adapter and high-level semantics $F_{hl}$ from GarmNet, fused via Garment Fusion blocks into the GarmDenoiser. It uses a coarse-to-fine training strategy that starts from VITON-HD data and then fine-tunes on a GarmRe-specific dataset to improve fidelity under occlusions. Quantitatively, IGR outperforms baselines on VITON-HD and StreetTryOn in metrics such as SSIM, LPIPS, DISTS, FID, CLIP-FID, and KID, while ablations confirm the utility of GarmNet, HQFT, and tuned guidance scale. This approach offers robust garment restoration suitable for downstream virtual try-on and fashion applications, leveraging strong diffusion priors to generate authentic garments aligned with reference persons.

Abstract

Garment restoration, the inverse of virtual try-on task, focuses on restoring standard garment from a person image, requiring accurate capture of garment details. However, existing methods often fail to preserve the identity of the garment or rely on complex processes. To address these limitations, we propose an improved diffusion model for restoring authentic garments. Our approach employs two garment extractors to independently capture low-level features and high-level semantics from the person image. Leveraging a pretrained latent diffusion model, these features are integrated into the denoising process through garment fusion blocks, which combine self-attention and cross-attention layers to align the restored garment with the person image. Furthermore, a coarse-to-fine training strategy is introduced to enhance the fidelity and authenticity of the generated garments. Experimental results demonstrate that our model effectively preserves garment identity and generates high-quality restorations, even in challenging scenarios such as complex garments or those with occlusions.

Figures (6)

• Figure 1: Several pairs of reference person image and our restored standard garment. From top to bottom, the pairs demonstrate the restoration of upper, lower, and full-body garments.
• Figure 2: Training data for GarmRe. The left part displays three types of data pairs that may lead to poor training. The right part showcases the data pairs perfectly matched for training.
• Figure 3: Overview of our proposed IGR. The GarmNet and GarmDenoiser share the same structural design. The reference person image, concatenated with the garment mask, serves as input to GarmNet, extracting high-level garment semantics. Simultaneously, the IP-Adapter captures low-level garment features from the segmented garment. The GarmDenoiser integrates garment features with intermediate outputs from GarmNet through self-attention layers, retaining only the latter half. Low-level features are incorporated via cross-attention layers for comprehensive garment reconstruction.
• Figure 4: Qualitative comparison. The first three rows are tested on the VITON-HD dataset, while the remaining rows are tested on the StreetTryOn dataset. GT refers the ground truth.
• Figure 5: Qualitative ablations for structure and training strategy.