Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval

Jian Jia, Jingtong Gao, Ben Xue, Junhao Wang, Qingpeng Cai, Quan Chen, Xiangyu Zhao, Peng Jiang, Kun Gai

TL;DR

The paper addresses the need for a unified understanding of discrete tokenizers across generation, comprehension, recommendation, and information retrieval, arguing that tokenization quality constrains LLM performance. It provides a systematic taxonomy and mechanism-level synthesis, detailing encoding, quantization, supervision, and backbone choices, plus a categorization of methods by modality and task. Key contributions include a unified framework for semantic tokenizers, a survey of state-of-the-art methods (including VQ-based, LFQ, FSQ, and RQVAE variants), and a rigorous discussion of challenges such as codebook collapse and cross-modal alignment. The work guides researchers and practitioners to design robust, scalable tokenization pipelines that leverage foundation models for multimodal AI.

Abstract

Discrete tokenizers have emerged as indispensable components in modern machine learning systems, particularly within the context of autoregressive modeling and large language models (LLMs). These tokenizers serve as the critical interface that transforms raw, unstructured data from diverse modalities into discrete tokens, enabling LLMs to operate effectively across a wide range of tasks. Despite their central role in generation, comprehension, and recommendation systems, a comprehensive survey dedicated to discrete tokenizers remains conspicuously absent in the literature. This paper addresses this gap by providing a systematic review of the design principles, applications, and challenges of discrete tokenizers. We begin by dissecting the sub-modules of tokenizers and systematically demonstrate their internal mechanisms to provide a comprehensive understanding of their functionality and design. Building on this foundation, we synthesize state-of-the-art methods, categorizing them into multimodal generation and comprehension tasks, and semantic tokens for personalized recommendations. Furthermore, we critically analyze the limitations of existing tokenizers and outline promising directions for future research. By presenting a unified framework for understanding discrete tokenizers, this survey aims to guide researchers and practitioners in addressing open challenges and advancing the field, ultimately contributing to the development of more robust and versatile AI systems.

From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval

TL;DR

The paper addresses the need for a unified understanding of discrete tokenizers across generation, comprehension, recommendation, and information retrieval, arguing that tokenization quality constrains LLM performance. It provides a systematic taxonomy and mechanism-level synthesis, detailing encoding, quantization, supervision, and backbone choices, plus a categorization of methods by modality and task. Key contributions include a unified framework for semantic tokenizers, a survey of state-of-the-art methods (including VQ-based, LFQ, FSQ, and RQVAE variants), and a rigorous discussion of challenges such as codebook collapse and cross-modal alignment. The work guides researchers and practitioners to design robust, scalable tokenization pipelines that leverage foundation models for multimodal AI.

Abstract

Discrete tokenizers have emerged as indispensable components in modern machine learning systems, particularly within the context of autoregressive modeling and large language models (LLMs). These tokenizers serve as the critical interface that transforms raw, unstructured data from diverse modalities into discrete tokens, enabling LLMs to operate effectively across a wide range of tasks. Despite their central role in generation, comprehension, and recommendation systems, a comprehensive survey dedicated to discrete tokenizers remains conspicuously absent in the literature. This paper addresses this gap by providing a systematic review of the design principles, applications, and challenges of discrete tokenizers. We begin by dissecting the sub-modules of tokenizers and systematically demonstrate their internal mechanisms to provide a comprehensive understanding of their functionality and design. Building on this foundation, we synthesize state-of-the-art methods, categorizing them into multimodal generation and comprehension tasks, and semantic tokens for personalized recommendations. Furthermore, we critically analyze the limitations of existing tokenizers and outline promising directions for future research. By presenting a unified framework for understanding discrete tokenizers, this survey aims to guide researchers and practitioners in addressing open challenges and advancing the field, ultimately contributing to the development of more robust and versatile AI systems.

Paper Structure

This paper contains 32 sections, 11 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Taxonomy of Discrete Tokenizer
  3. General Pipeline
  4. (1) Encoding.
  5. (2) Quantization.
  6. (3) Supervision.
  7. Backbone Network
  8. MLP-based.
  9. CNN-based.
  10. Transformer-based.
  11. Quantization Method
  12. Vanilla Quantization.
  13. Level-wise Quantization.
  14. Group-wise Quantization.
  15. Lookup-free Quantization.
  16. ...and 17 more sections

Figures (1)

  • Figure 1: Illustration of general pipeline and applications. Discrete tokenizers can be regarded as the interface between different input modalities and downstream tasks.