ControlEdit: A MultiModal Local Clothing Image Editing Method

TL;DR

ControlEdit tackles multimodal local clothing image editing by guiding diffusion-based inpainting with sketches, text, and masked source images. It relies on self-supervised learning to bypass the need for large paired datasets and introduces an inverse latent loss and latent-space mask fusion to maintain non-edited regions and ensure smooth boundaries, using Blended Latent Diffusion for natural transitions. The method extends ControlNet with a trainable branch and expanded conditioning channels to improve expressiveness. Experiments on the MGDbaldrati multimodal dataset show clear gains over baselines in fidelity, perceptual similarity, and semantic consistency, making it a strong baseline for future fashion editing tasks.

Abstract

Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions, which effectively improves the work efficiency of designers and reduces the threshold for user design. In this paper, we propose a new image editing method ControlEdit, which transfers clothing image editing to multimodal-guided local inpainting of clothing images. We address the difficulty of collecting real image datasets by leveraging the self-supervised learning approach. Based on this learning approach, we extend the channels of the feature extraction network to ensure consistent clothing image style before and after editing, and we design an inverse latent loss function to achieve soft control over the content of non-edited areas. In addition, we adopt Blended Latent Diffusion as the sampling method to make the editing boundaries transition naturally and enforce consistency of non-edited area content. Extensive experiments demonstrate that ControlEdit surpasses baseline algorithms in both qualitative and quantitative evaluations.

Figures (7)

• Figure 1: ControlEdit. Users can edit clothing by drawing conditional images.The first two rows of images are edited by regular users, while the last row is modified by professional fashion designers.
• Figure 2: ControlEdit Network Architecture.
• Figure 3: Masked Image Example.
• Figure 4: Image inference network structure.
• Figure 5: Qualitative comparison. SD Inpainting, Blended Latent Diffusion and Uni-paint synthesize clothing images driven by texts at the down side.