Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation

TL;DR

Control-CLIP tackles domain-specific style guidance in text-to-image generation by decoupling category and style representations within CLIP using two dedicated encoders ($f_s$, $f_c$) and a lightweight Adapter. It trains these encoders with labeled cross-entropy objectives and unlabeled triplet losses to disentangle style from category, and integrates their outputs into diffusion conditioning through a novel cross-attention mechanism. The approach demonstrates improved discrimination for both style and category across multiple datasets and enables more faithful generation to prompts with domain-specific styles via a Stable Diffusion plug-in, without modifying the generator. Overall, the method offers effective, data-efficient domain adaptation for text-to-image generation with preserved diversity and generation quality, suitable for broad deployment with minimal retraining.

Abstract

Text-to-image diffusion models have shown remarkable capabilities of generating high-quality images closely aligned with textual inputs. However, the effectiveness of text guidance heavily relies on the CLIP text encoder, which is trained to pay more attention to general content but struggles to capture semantics in specific domains like styles. As a result, generation models tend to fail on prompts like "a photo of a cat in Pokemon style" in terms of simply producing images depicting "a photo of a cat". To fill this gap, we propose Control-CLIP, a novel decoupled CLIP fine-tuning framework that enables the CLIP model to learn the meaning of category and style in a complement manner. With specially designed fine-tuning tasks on minimal data and a modified cross-attention mechanism, Control-CLIP can precisely guide the diffusion model to a specific domain. Moreover, the parameters of the diffusion model remain unchanged at all, preserving the original generation performance and diversity. Experiments across multiple domains confirm the effectiveness of our approach, particularly highlighting its robust plug-and-play capability in generating content with various specific styles.

Figures (4)

• Figure 1: Comparison of different approaches to leveraging in-domain text and image data for fine-tuning. Previously common fine-tuning methods include: (a) fine-tuning only the generator, (b) fine-tuning only the text encoder, and (c) using an adapter to decouple category features from the text encoder and combine them with the original text features. (d) Our Control-CLIP utilizes a decoupled design to learn both category and style features for generation models.
• Figure 2: Training style encoder in Control-CLIP. Left: On datasets with style labels, Control-CLIP separates category and style features through cross-entropy loss, achieving better performance but requires higher annotation requirements. Right: On datasets without style labels, we can utilize image captions to obtain style information. Control-CLIP uses a triplet loss function to distinguish between category and style features. The category encoder is trained similarly with an inverted loss function.
• Figure 3: We replace the input of the K and V matrices in the attention mechanism with the outputs of Style-Control and Category-Control CLIP.
• Figure 4: With the guidance of Control-CLIP, Stable Diffusion can generate images closely aligned with the prompts while maintaining the diversity of the generated contents. Compared to LoRA, Control-CLIP's generated results have better style and content diversity. Prompts: (a) A photo of Pokemon style cat; (b) A photo of Tom and Jerry style dog; (c) A photo of SpongeBob style tiger. For each cartoon domain, we use the same set for LoRA and Control-CLIP of 10 images for fine-tuning.