VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis

TL;DR

VersaGen addresses the limited visual control in text-to-image synthesis by enabling four flexible input modes (single subject, multiple subjects, background, or combinations) and integrating them into a frozen diffusion model via a trainable adaptor. It introduces three inference-time strategies—Multimodal Conflict Resolver, Multi-object Decoupling, and Adaptive Control Strength—to align user drawings with prompts and improve robustness in real-world use. Extensive experiments on COCO and Sketchy show VersaGen outperforms baselines in both qualitative and quantitative measures and receives strong user support for its ease of use. This work advances user-centric controllable generation by providing versatile, drawing-based conditioning that can handle diverse creative intents while maintaining scalability and practicality.

Abstract

Despite the rapid advancements in text-to-image (T2I) synthesis, enabling precise visual control remains a significant challenge. Existing works attempted to incorporate multi-facet controls (text and sketch), aiming to enhance the creative control over generated images. However, our pilot study reveals that the expressive power of humans far surpasses the capabilities of current methods. Users desire a more versatile approach that can accommodate their diverse creative intents, ranging from controlling individual subjects to manipulating the entire scene composition. We present VersaGen, a generative AI agent that enables versatile visual control in T2I synthesis. VersaGen admits four types of visual controls: i) single visual subject; ii) multiple visual subjects; iii) scene background; iv) any combination of the three above or merely no control at all. We train an adaptor upon a frozen T2I model to accommodate the visual information into the text-dominated diffusion process. We introduce three optimization strategies during the inference phase of VersaGen to improve generation results and enhance user experience. Comprehensive experiments on COCO and Sketchy validate the effectiveness and flexibility of VersaGen, as evidenced by both qualitative and quantitative results.

Figures (9)

• Figure 1: VersaGen can handle various forms of input provided by users, whether at the object-level, scene-level, or a combination of both.
• Figure 2: (a) The illustration of pilot study: users are tasked with generating an image similar to the given reference image using SD and ControlNet. (b) Quantitative evaluation of SD and ControlNet.
• Figure 3: The illustration of VersaGen. At denoising timestep $\tau$ during inference, MCR functions to update the original noisy latent $z_\tau$ to $z^{'}_\tau$, to alleviate potential conflicts across modalities.
• Figure 4: Visualised comparison of SD, T2I-Adapter, ControlNet, UniControl, GLIGEN, InstanceDiffusion and our proposed VersaGen. Problematic regions are highlighted with $\color{red}\Box$, and missing entities are indicated by $\color{red}?$.
• Figure 5: Visualisation of the reference images used in the human study alongside the user-submitted results generated by the three methods.