Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InfoTok: Regulating Information Flow for Capacity-Constrained Shared Visual Tokenization in Unified MLLMs

Lv Tang, Tianyi Zheng, Bo Li, Xingyu Li

TL;DR

Unified multimodal LLMs face conflicting demands from understanding and generation under a finite visual token budget. InfoTok reframes tokenization through the Information Bottleneck, regularizing the flow of information from images to shared tokens and to multimodal outputs via a variational IB formulation, yielding compact yet task-relevant representations. Empirical results across Harmon, OpenUni, and Show-o2 show consistent improvements in both understanding and generation without extra data, with ablations confirming the value of Compactness, Sufficiency, and Alignment. This work provides a principled foundation for learning a single, information-efficient token space in unified MLLMs and suggests a practical path for more interpretable, capacity-aware tokenization.

Abstract

Unified multimodal large language models (MLLMs) integrate image understanding and generation in a single framework, with the visual tokenizer acting as the sole interface that maps visual inputs into tokens for downstream tasks. However, existing shared-token designs are mostly architecture-driven and lack an explicit criterion for what information tokens should preserve to support both understanding and generation. Therefore, we introduce a capacity-constrained perspective, highlighting that in shared-token unified MLLMs the visual tokenizer behaves as a compute-bounded learner, so the token budget should prioritize reusable structure over hard-to-exploit high-entropy variations and redundancy. Motivated by this perspective, we propose InfoTok, an information-regularized visual tokenization mechanism grounded in the Information Bottleneck (IB) principle. InfoTok formulates tokenization as controlling information flow from images to shared tokens to multimodal outputs, yielding a principled trade-off between compression and task relevance via mutual-information regularization. We integrate InfoTok into three representative unified MLLMs without introducing any additional training data. Experiments show consistent improvements on both understanding and generation, supporting information-regularized tokenization as a principled foundation for learning a shared token space in unified MLLMs.

InfoTok: Regulating Information Flow for Capacity-Constrained Shared Visual Tokenization in Unified MLLMs

TL;DR

Unified multimodal LLMs face conflicting demands from understanding and generation under a finite visual token budget. InfoTok reframes tokenization through the Information Bottleneck, regularizing the flow of information from images to shared tokens and to multimodal outputs via a variational IB formulation, yielding compact yet task-relevant representations. Empirical results across Harmon, OpenUni, and Show-o2 show consistent improvements in both understanding and generation without extra data, with ablations confirming the value of Compactness, Sufficiency, and Alignment. This work provides a principled foundation for learning a single, information-efficient token space in unified MLLMs and suggests a practical path for more interpretable, capacity-aware tokenization.

Abstract

Unified multimodal large language models (MLLMs) integrate image understanding and generation in a single framework, with the visual tokenizer acting as the sole interface that maps visual inputs into tokens for downstream tasks. However, existing shared-token designs are mostly architecture-driven and lack an explicit criterion for what information tokens should preserve to support both understanding and generation. Therefore, we introduce a capacity-constrained perspective, highlighting that in shared-token unified MLLMs the visual tokenizer behaves as a compute-bounded learner, so the token budget should prioritize reusable structure over hard-to-exploit high-entropy variations and redundancy. Motivated by this perspective, we propose InfoTok, an information-regularized visual tokenization mechanism grounded in the Information Bottleneck (IB) principle. InfoTok formulates tokenization as controlling information flow from images to shared tokens to multimodal outputs, yielding a principled trade-off between compression and task relevance via mutual-information regularization. We integrate InfoTok into three representative unified MLLMs without introducing any additional training data. Experiments show consistent improvements on both understanding and generation, supporting information-regularized tokenization as a principled foundation for learning a shared token space in unified MLLMs.
Paper Structure (18 sections, 12 equations, 2 figures, 5 tables)

This paper contains 18 sections, 12 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Unified Multimodal Large Language Models
  5. Proposed Method
  6. Unified MLLM Frameworks
  7. Unified Information-Regularized Tokenization
  8. Classical IB Formulation
  9. Extended IB Formulation in InfoTok
  10. Variational IB Estimation in InfoTok
  11. Training Objective and Theoretical Grounding
  12. Training Objective
  13. Theoretical Grounding
  14. Experiment
  15. Experiment Setup
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Performance comparison of three representative unified MLLMs (Show-o2 xie2025show, OpenUni DBLP:journals/corr/abs-2505-23661 and Harmon DBLP:journals/corr/abs-2503-21979) before and after applying InfoTok regularization. Each radar chart reports results across representative image understanding benchmarks (UniBench DBLP:journals/corr/abs-2505-10483, GQA DBLP:conf/cvpr/HudsonM19, SEED DBLP:journals/corr/abs-2307-16125, POPE DBLP:conf/emnlp/LiDZWZW23, MME DBLP:journals/corr/abs-2306-13394, MMV2 DBLP:conf/icml/YuYLWL0WW24 and MMMU yue2024mmmu) and image generation benchmarks (Geneval DBLP:conf/nips/GhoshHS23, Geneval++ DBLP:journals/corr/abs-2508-09987 and WISE DBLP:journals/corr/abs-2503-07265). The InfoTok-regularized variants achieve consistently improved balance between semantic understanding and visual synthesis, demonstrating that information-theoretic regularization effectively enhances representation sufficiency and cross-modal generalization in unified multimodal learning. All InfoTok fine-tuning experiments are conducted without introducing any additional datasets, relying solely on the original training data used by each baseline.
  • Figure 2: Illustration of our information-regularized tokenization (InfoTok). (a) depicts a standard unified MLLM with shared tokenization, where a single visual tokenizer jointly supports understanding (I2T) and generation (T2I) tasks. (b) presents InfoTok, which imposes an information-regularization objective to explicitly control what visual content is retained in the shared token space. Specifically, InfoTok promotes compact yet sufficient tokens, filtering redundant information while preserving task-critical semantics for understanding and essential perceptual cues for generation. (c) illustrates the training workflow of InfoTok. During training, the Compactness, Sufficiency, and Alignment objectives form the InfoTok loss to regularize the shared visual tokens in existing unified MLLMs. Qualitatively, after applying InfoTok to Harmon and Show-o2, we observe visibly improved instruction following and higher overall generation quality.