EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation

Abstract

The development of unified multimodal large language models (MLLMs) is fundamentally challenged by the granularity gap between visual understanding and generation: understanding requires high-level semantic abstractions, while image generation demands fine-grained pixel-level representations. Existing approaches usually enforce the two supervision on the same set of representation or decouple these two supervision on separate feature spaces, leading to interference and inconsistency, respectively. In this work, we propose EvoTok, a unified image tokenizer that reconciles these requirements through a residual evolution process within a shared latent space. Instead of maintaining separate token spaces for pixels and semantics, EvoTok encodes an image into a cascaded sequence of residual tokens via residual vector quantization. This residual sequence forms an evolution trajectory where earlier stages capture low-level details and deeper stages progressively transition toward high-level semantic representations. Despite being trained on a relatively modest dataset of 13M images, far smaller than the billion-scale datasets used by many previous unified tokenizers, EvoTok achieves a strong reconstruction quality of 0.43 rFID on ImageNet-1K at 256x256 resolution. When integrated with a large language model, EvoTok shows promising performance across 7 out of 9 visual understanding benchmarks, and remarkable results on image generation benchmarks such as GenEval and GenAI-Bench. These results demonstrate that modeling visual representations as an evolving trajectory provides an effective and principled solution for unifying visual understanding and generation.

Figures (6)

• Figure 1: Different paradigms of current MLLMs and unified models. (a) Entangled unified MLLMs share semantic and pixel features, enabling both understanding and generation but tightly coupling the two objectives. (b) Decoupled unified MLLMs separate semantic and pixel encoders or feature layers to disentangle tasks. (c) Our unified evolution MLLM evolves pixel features into semantic representations within a shared space while keeping decoupled latents, achieving aligned understanding and generation while preserving decoupled representations.
• Figure 2: The overview of the proposed EvoTok. An image is encoded into a sequence of residual tokens within a shared latent space, forming a residual evolution trajectory. Tokens from earlier stages preserve pixel-level details for reconstruction, while tokens from deeper stages, formed by progressively accumulating tokens from full stages, capture semantic-level features aligned with those extracted by the pre-trained SigLIP2 tschannen2025siglip2. Within this residual evolution trajectory, low-level pixel features and high-level semantic representations co-evolve, enabling a decoupled and consistent representations to support both visual understanding and generation.
• Figure 3: Images reconstruction at a resolution of 256$\times$ 256 on ImageNet-1K.
• Figure 4: Images generated with our unified EvoTok at a resolution of 256$\times$ 256. Our model generates high-quality images spanning diverse visual domains, including portraits, landscapes, objects, artistic paintings, etc.
• Figure 5: Analysis of the Unified Latent Space.