Query-Kontext: An Unified Multimodal Model for Image Generation and Editing

TL;DR

Query-Kontext addresses the entanglement between multimodal reasoning and high-fidelity rendering in unified multimodal models by decoupling the reasoning performed by a multimodal LLM from the diffusion-based image synthesis. It introduces kontext tokens that carry semantic and coarse visual cues, acting as an interface between a VLM and a diffusion backbone through a three-stage progressive training pipeline. The approach uses an economical alignment strategy, a mixed data pipeline spanning real and synthetic data, and a low-level image encoder to enhance fidelity, achieving competitive GenEval and state-of-the-art-like results on editing benchmarks and multi-subject generation. This decoupled, scalable design improves efficiency and flexibility for image generation and editing tasks while enabling robust identity preservation and grounding across diverse multimodal prompts.

Abstract

Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I), whether instantiated as assembled unified frameworks which couple powerful vision-language model (VLM) with diffusion-based generator, or as naive Unified Multimodal Models with an early fusion of understanding and generation modalities. We contend that in current unified frameworks, the crucial capability of multimodal generative reasoning which encompasses instruction understanding, grounding, and image referring for identity preservation and faithful reconstruction, is intrinsically entangled with high-fidelity synthesis. In this work, we introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal ``kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs. This design delegates the complex ability of multimodal generative reasoning to powerful VLM while reserving diffusion model's role for high-quality visual synthesis. To achieve this, we propose a three-stage progressive training strategy. First, we connect the VLM to a lightweight diffusion head via multimodal kontext tokens to unleash the VLM's generative reasoning ability. Second, we scale this head to a large, pre-trained diffusion model to enhance visual detail and realism. Finally, we introduce a low-level image encoder to improve image fidelity and perform instruction tuning on downstream tasks. Furthermore, we build a comprehensive data pipeline integrating real, synthetic, and open-source datasets, covering diverse multimodal reference-to-image scenarios, including image generation, instruction-driven editing, customized generation, and multi-subject composition. Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.

Figures (10)

• Figure 2: The overall framework of the unified multi-modal to image generation and editing model, Query-Kontext.
• Figure 3: Three training stages of Query-Kontext. Note that the Diffusion Head is only used in the Stage 1. In the Stage 2 and 3, we scale up Diffusion model to 10× parameters and keep MLLM frozen to provide coarse-grained image conditions.
• Figure 4: Examples of the image transformation task. Each row shows a transformation instruction, the source image and the resulting target image, in order from left to right.
• Figure 5: Example of the prompt used in Qwen2.5-VL to generate open-ended transformation instructions.
• Figure 6: Examples of synthetic data pipeline for instruction Editing.