ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning

TL;DR

The paper tackles the semantic gap between sparse text prompts and target images in controllable image generation by introducing ControlThinker, a realize-then-generate framework that treats an MLLM as a visual reasoning engine to enrich prompts from control signals. A dedicated ControlThink-50K dataset enables supervised and reinforcement fine-tuning (SFT and RFT) to generate dense, reasoning-based prompts; inference-time scaling further selects the best reasoning trajectory using dual rewards. Without modifying the underlying generator, ControlThinker achieves state-of-the-art layout consistency and image fidelity across multiple control types, outperforming ControlNet, ControlNet++, and ControlAR on both structural and perceptual metrics. The approach demonstrates practical impact by leveraging pre-trained models to bridge semantic gaps, enabling more accurate and coherent controllable image synthesis. The code and models are publicly available, promoting adoption and further development in controllable generation research.

Abstract

The field of controllable image generation has seen significant advancements, with various architectures improving generation layout consistency with control signals. However, contemporary methods still face challenges in bridging the semantic gap between input text prompts with sparse semantics and the target images, often over-relying on low-level control signals to infer regional details. To address this challenge, we propose ControlThinker, a novel framework that employs a "comprehend-then-generate" paradigm. Firstly, by incentivizing the visual reasoning capability of a MLLM, latent semantics from control images are mined to enrich text prompts. This enriched semantic understanding then seamlessly aids in image generation without the need for additional complex modifications. To further tackle the uncertainty arising from the ambiguity of control images, we encourage broader exploration of reasoning trajectories and select the optimal one using a metric-based output reward model (ORM). Extensive experimental results demonstrate that ControlThinker effectively mitigates the semantic gap between raw text prompts and target images, resulting in improved visual quality and semantic consistency across a wide range of benchmarks. The code and models are available at https://github.com/Maplebb/ControlThinker.

Figures (12)

• Figure 1: Comparison between our ControlThinker and previous pipelines. Unlike previous methods that struggle to generate images adhering to the text prompt, ControlThinker leverages an MLLM as an input comprehender to enrich text semantics, enabling more accurate generation.
• Figure 2: The SFT and RFT two stage training paradigm and Inference Time Scaling method of our ControlThinker.
• Figure 3: Comparison of enhanced prompts from different models including zero shot model, model after SFT, and model after SFT and RFT.
• Figure 4: The Data Construction Pipeline of ControlThinker.
• Figure 5: Qualitative comparison with state-of-the-art approaches across different condition types.