Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

TL;DR

Bifrost-1 introduces a practical bridge between pretrained multimodal LLMs and diffusion models by using patch-level CLIP latents that align with the MLLM's visual encoder. It adds a trainable visual generation branch and a Latent ControlNet to preserve the MLLM's multimodal reasoning while enabling high-fidelity image synthesis with decoupled training for efficiency. Empirical results show competitive visual generation quality and strong multimodal understanding with substantially lower training compute than prior unified or bridging approaches. The approach offers a scalable, resource-efficient path toward unified multimodal models that integrate reasoning with high-quality visual generation.

Abstract

There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MLLM's CLIP visual encoder. These patch-level image embeddings are integrated into the diffusion model with a lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices.

Figures (8)

• Figure 1: Comparison of different approaches in using LLM for image generation. (a) Single architecture handling both text and image tokens into the LLM/MLLM. (b) Bridging LLM and Diffusion model with a 1D sequence (image tokens or text tokens). (c) Bifrost-1 (ours), which bridges MLLM with diffusion models with 2D image tokens aligned with MLLM embeddings.
• Figure 2: Overview of Bifrost-1. Bifrost-1 equips the backbone MLLM with a visual generation branch, which is a trainable copy of a pretrained MLLM parameters (i.e., QKV, MLP, normalization layers) and a newly added vision head (i.e., a linear layer). The visual generation branch outputs patch-level CLIP latents, which are then downsampled and reshaped into 2D (HxW), provided to latent ControlNet, and finally guiding image generation of a pretrained diffusion model. During training, a portion of the image patches is randomly replaced with learnable mask tokens <M>. During inference, we start with fully masked image tokens and autoregressively predict them.
• Figure 3: Attention mask. X-axis is input and Y-axis is output.
• Figure 4: Image reconstruction scores with different numbers of 2D CLIP latent tokens used within Bifrost-1 on ImageNet for around one training epoch ($\sim$26K steps). Results indicate that using more tokens achieves better data efficiency.
• Figure 5: Visual samples for image reconstruction with different numbers of patch-level CLIP tokens generated from MLLM. The Latent ControlNet models with varying numbers of tokens are trained for only 1 epoch on the ImageNet training split.