Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-to-End Vision Tokenizer Tuning

Wenxuan Wang, Fan Zhang, Yufeng Cui, Haiwen Diao, Zhuoyan Luo, Huchuan Lu, Jing Liu, Xinlong Wang

TL;DR

This work tackles the bottleneck created by decoupled vision tokenizers in multimodal learning by introducing End-to-End Vision Tokenizer Tuning (ETT). ETT makes the tokenizer trainable via its codebook embeddings and connects it to a pretrained LLM through a lightweight projector, enabling joint end-to-end optimization with both captioning and reconstruction objectives. Through a three-stage training pipeline and extensive ablations, ETT yields consistent improvements in multimodal understanding and generation while largely preserving the tokenizer's reconstruction ability, achieving strong results with smaller models and data footprints. The approach offers a simple, practical pathway to more effective multimodal foundation models and suggests extensions to broader modalities in future work.

Abstract

Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.

End-to-End Vision Tokenizer Tuning

TL;DR

This work tackles the bottleneck created by decoupled vision tokenizers in multimodal learning by introducing End-to-End Vision Tokenizer Tuning (ETT). ETT makes the tokenizer trainable via its codebook embeddings and connects it to a pretrained LLM through a lightweight projector, enabling joint end-to-end optimization with both captioning and reconstruction objectives. Through a three-stage training pipeline and extensive ablations, ETT yields consistent improvements in multimodal understanding and generation while largely preserving the tokenizer's reconstruction ability, achieving strong results with smaller models and data footprints. The approach offers a simple, practical pathway to more effective multimodal foundation models and suggests extensions to broader modalities in future work.

Abstract

Existing vision tokenization isolates the optimization of vision tokenizers from downstream training, implicitly assuming the visual tokens can generalize well across various tasks, e.g., image generation and visual question answering. The vision tokenizer optimized for low-level reconstruction is agnostic to downstream tasks requiring varied representations and semantics. This decoupled paradigm introduces a critical misalignment: The loss of the vision tokenization can be the representation bottleneck for target tasks. For example, errors in tokenizing text in a given image lead to poor results when recognizing or generating them. To address this, we propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks. Unlike prior autoregressive models that use only discrete indices from a frozen vision tokenizer, ETT leverages the visual embeddings of the tokenizer codebook, and optimizes the vision tokenizers end-to-end with both reconstruction and caption objectives. ETT can be seamlessly integrated into existing training pipelines with minimal architecture modifications. Our ETT is simple to implement and integrate, without the need to adjust the original codebooks or architectures of the employed large language models. Extensive experiments demonstrate that our proposed end-to-end vision tokenizer tuning unlocks significant performance gains, i.e., 2-6% for multimodal understanding and visual generation tasks compared to frozen tokenizer baselines, while preserving the original reconstruction capability. We hope this very simple and strong method can empower multimodal foundation models besides image generation and understanding.
Paper Structure (13 sections, 3 equations, 3 figures, 5 tables)

This paper contains 13 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Vision Tokenizer
  5. End-to-End Vision Tokenizer Tuning
  6. Training Recipe for Multimodal Generation and Understanding
  7. Experimental Results
  8. Training Settings
  9. Multimodal Understanding Evaluation
  10. Visual Generation Evaluation
  11. Ablation Study
  12. Conclusion
  13. Limitations and Future Topics

Figures (3)

  • Figure 1: Left: Existing autoregressive pipeline uses the discrete indices from a frozen vision tokenizer optimized with low-level reconstruction. Middle: We present ETT, an end-to-end tokenizer tuning approach which takes advantage of the visual codebook embeddings and optimizes the vision tokenizer and downstream training jointly. Right: Our proposed ETT unlocks significant performance gains on multimodal understanding and generation benchmarks.
  • Figure 2: Visual generation results with our ETT. We present 512 × 512 results spanning different styles, subjects, and scenarios. Note that the presented prompts are simplified versions, which convey the general meaning.
  • Figure 3: Comparison of visual reconstruction results of the input image before and after end-to-end tuning by our ETT. The vision tokenizer tuned by our ETT benefits from retaining the original rich low-level detail representations while being effectively injected with high-level semantics, producing visual details comparable to the pre-tuned counterpart and even performs better in some detail reconstruction, e.g., text rendering.