MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer

TL;DR

Manzano addresses the trade-off between image understanding and generation in unified multimodal LLMs by introducing a unified hybrid image tokenizer and a shared autoregressive backbone. The model uses continuous tokens for image-to-text understanding and discrete tokens for text-to-image generation, both mapped into a common semantic space, with a diffusion-based image decoder for pixel synthesis. Through a three-stage training recipe combining large-scale text data, IT/TI data, and instruction fine-tuning, Manzano achieves state-of-the-art performance among unified models and competitive results with specialist systems, especially on text-rich tasks. Scaling experiments show monotonic improvements with larger LLM decoders and diffusion decoders, and ablations demonstrate minimal cross-task interference, validating the design decisions around the hybrid tokenizer and integrated AR framework. The work also demonstrates effective image editing capabilities by conditioning both the LLM and diffusion decoder on a reference image, suggesting practical applicability in multimodal reasoning and creative generation.

Abstract

Unified multimodal Large Language Models (LLMs) that can both understand and generate visual content hold immense potential. However, existing open-source models often suffer from a performance trade-off between these capabilities. We present Manzano, a simple and scalable unified framework that substantially reduces this tension by coupling a hybrid image tokenizer with a well-curated training recipe. A single shared vision encoder feeds two lightweight adapters that produce continuous embeddings for image-to-text understanding and discrete tokens for text-to-image generation within a common semantic space. A unified autoregressive LLM predicts high-level semantics in the form of text and image tokens, with an auxiliary diffusion decoder subsequently translating the image tokens into pixels. The architecture, together with a unified training recipe over understanding and generation data, enables scalable joint learning of both capabilities. Manzano achieves state-of-the-art results among unified models, and is competitive with specialist models, particularly on text-rich evaluation. Our studies show minimal task conflicts and consistent gains from scaling model size, validating our design choice of a hybrid tokenizer.

Figures (9)

• Figure 1: Qualitative text-to-image generation on challenging prompts. Manzano handles counterintuitive, physics-defying prompts (e.g., ‘The bird is flying below the elephant’) comparably to GPT-4o islam2024gpt and Nano Banana google2025nano.
• Figure 2: Quantitative comparisons on popular understanding and generation benchmarks. Manzano 3B and 30B models achieve superior or competitive performance compared to other SOTA unified multimodal LLMs.
• Figure 3: Our hybrid tokenizer workflow. (Left): The tokenizer produces two distinct but homogeneous feature streams through separate adapters. During training, one adapter output is randomly sampled and passed to a small LLM decoder for alignment. (Right): Once the tokenizer is trained, the right panel illustrates how these two feature types are applied to understanding and generation tasks.
• Figure 4: Training overview. (Left): Unified LLM training with hybrid tokens, the continuous adapter produces embeddings used for the text loss, while the discrete adapter generates hard tokens serving as targets for the image loss. (Right): With vision encoder and adapters fixed, an image decoder is trained to reconstruct images using a diffusion loss.
• Figure 5: Unified vs. Single-task study. Our unified model exhibits a slight regression compared with the understanding-only model on understanding task; however, this effect becomes negligible at the 3B scale, where the gap is less than 1.0. For generation, the unified model shows a decline on only one benchmark compared with the generation-only model.