RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilation

TL;DR

RAGDiffusion introduces a Retrieval-Augmented Generation framework for faithful cloth generation, integrating a dual-tower StructureNet with a retrieval-based Structure Locally Linear Embedding (SLLE) and an omni-level texture alignment pipeline. By coupling soft/hard structure guidance from retrieved embeddings and silhouette landmarks with a coarse-to-fine generation backbone (ReferenceNet and PGEA), it achieves higher structural integrity and texture fidelity on challenging in-the-wild garment data. The approach demonstrates strong improvements over baselines, offers zero-shot generalization via retrieval database expansion, and enables human-interpretable control through landmark manipulation, with practical implications for e-commerce, product design, and virtual fitting. Overall, RAGDiffusion advances high-specification faithful generation by leveraging external knowledge to counter intrinsic hallucinations in garment synthesis.

Abstract

Standard clothing asset generation involves restoring forward-facing flat-lay garment images displayed on a clear background by extracting clothing information from diverse real-world contexts, which presents significant challenges due to highly standardized structure sampling distributions and clothing semantic absence in complex scenarios. Existing models have limited spatial perception, often exhibiting structural hallucinations and texture distortion in this high-specification generative task. To address this issue, we propose a novel Retrieval-Augmented Generation (RAG) framework, termed RAGDiffusion, to enhance structure determinacy and mitigate hallucinations by assimilating knowledge from language models and external databases. RAGDiffusion consists of two processes: (1) Retrieval-based structure aggregation, which employs contrastive learning and a Structure Locally Linear Embedding (SLLE) to derive global structure and spatial landmarks, providing both soft and hard guidance to counteract structural ambiguities; and (2) Omni-level faithful garment generation, which introduces a coarse-to-fine texture alignment that ensures fidelity in pattern and detail components within the diffusing. Extensive experiments on challenging real-world datasets demonstrate that RAGDiffusion synthesizes structurally and texture-faithful clothing assets with significant performance improvements, representing a pioneering effort in high-specification faithful generation with RAG to confront intrinsic hallucinations and enhance fidelity.

Figures (19)

• Figure 1: RAGDiffusion synthesizes structurally and pattern-wise faithful standard garments by assimilating retrieved knowledge.
• Figure 2: Overall framework of RAGDiffusion. StructureNet provides latent structural embeddings, while SLLE facilitates the embedding fusion along with landmark retrieval. The generative model synthesizes multiple conditions to achieve omni-level high-fidelity generation.
• Figure 3: RAGDiffusion delivers both loyal structures and superior details on challenging layered and side-view situations.
• Figure 4: Boxplot illustration of user study. RAGDiffusion demonstrates better performance and stability across scenes.
• Figure 5: Ablation study. Without landmarks, RAGDiffusion suffers inaccurate shape. Embeddings from StructureNet improve inner structure, while PGEA enhances detail preservation.