Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation

TL;DR

Text prompts in diffusion-based image generation often fail to precisely capture user intent, especially with ambiguous language. The authors propose Cocktail, a pipeline that fuses multiple modalities into a single model via gControlNet, Controllable Normalisation, and Spatial Guidance Sampling to achieve multi-modal and spatially refined control. The key contributions are the Generalized ControlNet, ControlNorm, and spatial guidance strategy, with extensive ablations and COCO5k experiments showing improved fidelity and alignment over baselines. This work enables flexible, region-aware generation with a single, scalable model, though it notes limitations in spatial guidance ease-of-use and broader ethical considerations.

Abstract

Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals.

Figures (15)

• Figure 1: Comparison of various control methods. Our approach requires only one generalized model, unlike previous that needed multiple models for multiple modalities.
• Figure 2: Examples of our model with the same prompt. Given a text prompt along with various modality signals, our approach is able to synthesize images that satisfy all input conditions or any arbitrary subset of these conditions using a single model. The prompt is: A girl holding a cat.
• Figure 3: The network architecture of Generalized ControlNet (gControlNet) with Controllable Normalisation (ControlNorm). The parameters indicated by the yellow sections are sourced from the pre-trained model and stay constant, while only those in the blue sections are updated during training, with the gradient back-propagated along the blue arrows.
• Figure 4: Our model can generate images with the provided prompts and multi-modality information (e.g., edge, pose, and segmentation map) across various scales.
• Figure 5: Qualitative comparison of Uni-Modality on the COCO validation set.