Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification

TL;DR

This paper argues that compression is a core driver of intelligence by linking classical visual coding with emergent visual token technologies used in multimodal large models. It proposes a unified information-theoretic framework that juxtaposes Shannon entropy for pixel-level coding with semantic entropy for token-based reasoning, and contrasts rate-distortion with the information bottleneck as core optimization lenses. Through a systematic survey and theoretical synthesis, the work derives bidirectional insights: classical coding principles can refine tokenization, and token-based semantics can guide next-generation codecs tailored for machine tasks. The practical implications span MLLMs, AIGC, and embodied AI, suggesting a path toward unified tokenizers and potentially standardizing token-based representations akin to traditional codecs.

Abstract

"Compression Tells Intelligence", is supported by research in artificial intelligence, particularly concerning (multimodal) large language models (LLMs/MLLMs), where compression efficiency often correlates with improved model performance and capabilities. For compression, classical visual coding based on traditional information theory has developed over decades, achieving great success with numerous international industrial standards widely applied in multimedia (e.g., image/video) systems. Except that, the recent emergingvisual token technology of generative multi-modal large models also shares a similar fundamental objective like visual coding: maximizing semantic information fidelity during the representation learning while minimizing computational cost. Therefore, this paper provides a comprehensive overview of two dominant technique families first -- Visual Coding and Vision Token Technology -- then we further unify them from the aspect of optimization, discussing the essence of compression efficiency and model performance trade-off behind. Next, based on the proposed unified formulation bridging visual coding andvisual token technology, we synthesize bidirectional insights of themselves and forecast the next-gen visual codec and token techniques. Last but not least, we experimentally show a large potential of the task-oriented token developments in the more practical tasks like multimodal LLMs (MLLMs), AI-generated content (AIGC), and embodied AI, as well as shedding light on the future possibility of standardizing a general token technology like the traditional codecs (e.g., H.264/265) with high efficiency for a wide range of intelligent tasks in a unified and effective manner.

Figures (12)

• Figure 1: The overall organization of this paper.
• Figure 2: A taxonomy of modern video coding paradigms, categorized by their different optimization objectives. The left branch represents traditional and neural codecs optimized for pixel fidelity (e.g., JPEG wallace1991jpeg, PNG boutell1997png, HEVC bross2013hevc, VVC bross2021vvc, DVC lu2019dvc, DCVC li2021deep_dcvc, etc). The right branch focuses on coding for human perception (e.g., PerCo careil2024towards, Diffeic li2024towards, DiffC theis2022lossy, MS-ILLM muckley2023improving) and coding for machine tasks (e.g., Channel Selection liu2022improving, TransTIC chen2023transtic, Adapt-ICMH li2024image).
• Figure 3: Pipeline of (visual) token technology, typically used in the mainstream (multi-modal) large language models (LLMs/MLLMs). Visual inputs are first converted into visual tokens by a visual tokenizer, which may be either continuous (patchify + linear projection with positional encoding, as in CLIP/SigLIP/DINOv2) or discrete (latent encoding and codebook quantization, as in VQ-VAE/VQ-GAN), thereby forming transformer-ready sequences. A subsequent visual token compression stage (e.g., attention-, similarity-, query-, pooling-, or RL-based) reduces the visual tokens to a small budget that, together with text tokens, feeds the token reasoning module for cross-modal fusion and inference. Arrows indicate the data flow from tokenization to compression, then reasoning.
• Figure 4: Six–axis view of visual token compression. The center Goal (acceleration, memory/KV reduction, long-context) is realized by choices along five orthogonal axes: Methodology (attention, similarity, RL, query, transformation), Training pattern (TF, Post, E2E, RL), Location (encoder, bridge, LM/KV, hybrid), Guidance (vision, text, hybrid), and Schedule (static, one-shot, dynamic, progressive). Arrows indicate how these factors compose into a concrete compression policy under a fixed visual budget $K$; the taxonomy matches Sec. \ref{['sec:compression']} and Table \ref{['tab:vtoken-15x7-categorical']}.
• Figure 5: Comparison of Discrete vs. Continuous Image Tokenization Paradigms. (A) Discrete Tokenization: The input image is encoded into dense vectors and then quantized using a learnable codebook to produce discrete indices ($z_q$). These tokens are processed by a discrete prior model (e.g., AR yu2024image Transformer vaswani2017attention). (B) Continuous Tokenization: The encoder maps the image directly to continuous latent variables ($z$) without quantization. These latents are modeled by a continuous prior, such as a Diffusion Model ho2020denoising or Flow Matching lipman2022flow. Both frameworks utilize conditioning inputs (e.g., text) and a decoder for image reconstruction.