Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing

TL;DR

This paper tackles the task of multimodal-conditioned fashion image editing and extends two existing fashion datasets, Dress Code and VITON-HD, with multimodal annotations, to generate human-centric fashion images guided by multimodal prompts.

Abstract

Fashion illustration is a crucial medium for designers to convey their creative vision and transform design concepts into tangible representations that showcase the interplay between clothing and the human body. In the context of fashion design, computer vision techniques have the potential to enhance and streamline the design process. Departing from prior research primarily focused on virtual try-on, this paper tackles the task of multimodal-conditioned fashion image editing. Our approach aims to generate human-centric fashion images guided by multimodal prompts, including text, human body poses, garment sketches, and fabric textures. To address this problem, we propose extending latent diffusion models to incorporate these multiple modalities and modifying the structure of the denoising network, taking multimodal prompts as input. To condition the proposed architecture on fabric textures, we employ textual inversion techniques and let diverse cross-attention layers of the denoising network attend to textual and texture information, thus incorporating different granularity conditioning details. Given the lack of datasets for the task, we extend two existing fashion datasets, Dress Code and VITON-HD, with multimodal annotations. Experimental evaluations demonstrate the effectiveness of our proposed approach in terms of realism and coherence concerning the provided multimodal inputs.

Figures (9)

• Figure 1: Example of images generated using the proposed Textual-inverted Multimodal Garment Designer (Ti-MGD) method, with each row featuring the same model edited using different inputs. For each generated image, we show the generation input conditions: texture (top left), keypoints (middle left), sketch (bottom left), and text (bottom of each column).
• Figure 2: Overview of the proposed Textual-inverted Multimodal Garment Designer (Ti-MGD) approach, a human-centric latent diffusion model conditioned on multiple modalities, including text, human pose, garment sketch, and fabric texture. The denoising UNet $\epsilon_{\theta}$ takes as input the latent variable $z_T$ and the spatial conditioning inputs (i.e. encoded masked model $\mathcal{E}(I_M)$, inpainting mask $m$, body keypoints $p$, and encoded sketch $\mathcal{E}(\bar{S})$). We incorporate text conditioning $Y$ using Stable Diffusion cross-attention capabilities, extending this mechanism to condition the generated image on the texture image $X$ by projecting it into the CLIP pseudo-word token embedding space. For this purpose, we utilize distinct cross-attention layers dedicated to text and texture conditioning.
• Figure 3: Detail of cross-attention layers of the denoising network, that are categorized into four groups based on spatial resolution. Group 3 contains the highest-resolution layers, while Group 0 comprises the lowest-resolution ones.
• Figure 4: Sample images and multimodal data from our newly collected Dress Code Multimodal and VITON-HD Multimodal datasets.
• Figure 5: Qualitative comparison of images generated using our approach with SDv1 (Ti-MGD) versus ControlNet with IP-Adapter. The smaller images represent the model inputs, while the bigger images depict the generated outputs.