UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model

TL;DR

UniWeTok introduces a unified discrete tokenizer with a massive binary codebook of $2^{128}$ to bridge high-fidelity reconstruction, semantic extraction, and generative capability in Unified Multimodal LLMs. It combines a SigLu-enabled convolution-attention hybrid architecture with Pre-Post Distillation and Generative-Aware Prior in a three-stage curriculum to achieve robust performance across resolutions and perception-sensitive content. Empirically, it attains state-of-the-art or competitive results on image generation, understanding, and editing benchmarks while demanding substantially lower training compute than peers. This work provides a strong baseline for future Unified MLLMs and emphasizes the viability of a single, well-optimized tokenizer to unify perception, reasoning, and generation across modalities.

Abstract

Unified Multimodal Large Language Models (MLLMs) require a visual representation that simultaneously supports high-fidelity reconstruction, complex semantic extraction, and generative suitability. However, existing visual tokenizers typically struggle to satisfy these conflicting objectives within a single framework. In this paper, we introduce UniWeTok, a unified discrete tokenizer designed to bridge this gap using a massive binary codebook ($\mathit{2^{128}}$). For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens. In terms of model architecture, we propose a convolution-attention hybrid architecture with the SigLu activation function. SigLu activation not only bounds the encoder output and stabilizes the semantic distillation process but also effectively addresses the optimization conflict between token entropy loss and commitment loss. We further propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios, such as those involving human faces and textual content. On ImageNet, UniWeTok achieves state-of-the-art image generation performance (FID: UniWeTok 1.38 vs. REPA 1.42) while requiring a remarkably low training compute (Training Tokens: UniWeTok 33B vs. REPA 262B). On general-domain, UniWeTok demonstrates highly competitive capabilities across a broad range of tasks, including multimodal understanding, image generation (DPG Score: UniWeTok 86.63 vs. FLUX.1 [Dev] 83.84), and editing (GEdit Overall Score: UniWeTok 5.09 vs. OmniGen 5.06). We release code and models to facilitate community exploration of unified tokenizer and MLLM.

Figures (9)

• Figure 1: Various resolution samples generated by our Unified MLLM based on UniWeTok, showcasing its capabilities in prompt adherence, spatial reasoning, and text rendering across various artistic styles. UniWeTok employs a spatial downsampling rate of 32$\times$ and a codebook size of $2^{128}$, which reduces the number of visual tokens by 75% while maintaining exceptionally high reconstruction quality. Furthermore, UniWeTok possesses strong semantic extraction capabilities and generative priors, making compressed discrete visual tokens suitable for Unified Multimodal Large Language Model.
• Figure 2: Illustration of UniWeTok training framework. We introduce a pre-trained semantic encoder for Pre-Post Distillation and a lightweight generative model for Generative-Aware Prior.
• Figure 3: Illustration of the three-stage training pipeline of UniWeTok.
• Figure 4: Ablation study of three-stage training pipeline.
• Figure 5: Detail Illustration of the UniWeTok model architecture.