UniFusion: Vision-Language Model as Unified Encoder in Image Generation

TL;DR

UniFusion introduces a diffusion-based image generator that uses a frozen Vision-Language Model as a unified encoder. The key innovations are Layerwise Attention Pooling (LAP), which aggregates multi-layer VLM features to condition the diffusion model, and VERIFI, a rewriting-based conditioning mechanism that leverages the VLM’s reasoning without re-encoding. The authors demonstrate strong text-to-image generation and editing performance with a single model, achieving competitive results against larger, post-trained baselines while exhibiting notable zero-shot generalization across unseen modalities and multi-reference inputs. The work highlights practical design choices (layer selection, refiners, and inference strategies) and provides insights into benchmark reliability, arguing for refined evaluation protocols. Overall, UniFusion shows that a unified VLM-based encoder can support high-fidelity generation, editing, and cross-task capabilities with efficient training and transferability across VLM families.

Abstract

Although recent advances in visual generation have been remarkable, most existing architectures still depend on distinct encoders for images and text. This separation constrains diffusion models' ability to perform cross-modal reasoning and knowledge transfer. Prior attempts to bridge this gap often use the last layer information from VLM, employ multiple visual encoders, or train large unified models jointly for text and image generation, which demands substantial computational resources and large-scale data, limiting its accessibility.We present UniFusion, a diffusion-based generative model conditioned on a frozen large vision-language model (VLM) that serves as a unified multimodal encoder. At the core of UniFusion is the Layerwise Attention Pooling (LAP) mechanism that extracts both high level semantics and low level details from text and visual tokens of a frozen VLM to condition a diffusion generative model. We demonstrate that LAP outperforms other shallow fusion architectures on text-image alignment for generation and faithful transfer of visual information from VLM to the diffusion model which is key for editing. We propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI), which conditions a diffusion transformer (DiT) only on the text tokens generated by the VLM during in-model prompt rewriting. VERIFI combines the alignment of the conditioning distribution with the VLM's reasoning capabilities for increased capabilities and flexibility at inference. In addition, finetuning on editing task not only improves text-image alignment for generation, indicative of cross-modality knowledge transfer, but also exhibits tremendous generalization capabilities. Our model when trained on single image editing, zero-shot generalizes to multiple image references further motivating the unified encoder design of UniFusion.

Figures (24)

• Figure 1: Diverse text-to-image generation with UniFusion. (Zoom in for more details)
• Figure 2: Diverse textual image editing and image reference workflows with UniFusion. All images encoded by VLM features only, no VAE tokens involved. (Zoom in for more details)
• Figure 3: Zero-shot capabilities by UniFusion, which was not explicitly trained for. Our unified encoder setup enables the transfer of many capabilities from the VLM encoder to generative image applications.
• Figure 4: UniFusion architecture and inference paradigm. We extract multimodal representations from multiple layers of a frozen LLM and aggregate with a learnable layerwise attention pooling (LAP) module. A subsequent refiner counteracts the VLM's position bias due to causal attention. VLM-Enabled Rewriting Injection with Flexible Inference (Verifi) rewrites the original input in-context. The rewritten tokens used for DiT conditioning leverage the VLM's reasoning capabilities to contextualize the target scene into a unified representation.
• Figure 5: Overview of considered architectures for unified VLM conditioning. Blue blocks within the VLM and DiT module denote selected layers, red denotes trainable modules, and gray denotes frozen modules.