TokBench: Evaluating Your Visual Tokenizer before Visual Generation

TL;DR

TokBench addresses the gap between current visual tokenizers/VAEs and the practical upper bound of visual generation quality by focusing on two human-sensitive contents: text and faces. It introduces a lightweight benchmark and Metrics for Text (T-ACC, T-NED) and Faces (F-Sim), plus a diverse dataset of images and videos, enabling efficient evaluation on standard hardware. The findings show that many discrete tokenizers lag at small scales, while modern continuous VAEs can reach higher fidelity, yet traditional metrics like FID or PSNR fail to capture these domain-specific reconstructions; TokBench metrics provide complementary insight and better alignment with human judgments. The work extends to video, revealing similar trends and highlighting the continued challenge of preserving fine-grained, small-scale information across modalities, with implications for designing upper-bound-aware visual generation systems and evaluating future tokenization schemes.

Abstract

In this work, we reveal the limitations of visual tokenizers and VAEs in preserving fine-grained features, and propose a benchmark to evaluate reconstruction performance for two challenging visual contents: text and face. Visual tokenizers and VAEs have significantly advanced visual generation and multimodal modeling by providing more efficient compressed or quantized image representations. However, while helping production models reduce computational burdens, the information loss from image compression fundamentally limits the upper bound of visual generation quality. To evaluate this upper bound, we focus on assessing reconstructed text and facial features since they typically: 1) exist at smaller scales, 2) contain dense and rich textures, 3) are prone to collapse, and 4) are highly sensitive to human vision. We first collect and curate a diverse set of clear text and face images from existing datasets. Unlike approaches using VLM models, we employ established OCR and face recognition models for evaluation, ensuring accuracy while maintaining an exceptionally lightweight assessment process <span style="font-weight: bold; color: rgb(214, 21, 21);">requiring just 2GB memory and 4 minutes</span> to complete. Using our benchmark, we analyze text and face reconstruction quality across various scales for different image tokenizers and VAEs. Our results show modern visual tokenizers still struggle to preserve fine-grained features, especially at smaller scales. We further extend this evaluation framework to video, conducting comprehensive analysis of video tokenizers. Additionally, we demonstrate that traditional metrics fail to accurately reflect reconstruction performance for faces and text, while our proposed metrics serve as an effective complement.

Figures (8)

• Figure 1: Comparison of Different Metrics with Human Judgments. In each case, previous metrics (PSNR, SSIM, LPIPS) demonstrate discrepancies with human assessments, whereas our proposed face similarity and text accuracy effectively reflect the reconstruction quality. The reference image represents the original, while Patch 0 and Patch 1 show reconstruction results from different visual tokenizers. The same regions are cropped from the complete images for visualization.
• Figure 2: Statistics and Sample Diversity of TokBench-Image. TokBench features a balanced instance-scale distribution with particular emphasis on small-scale face and text instances, presenting significant challenges for existing visual reconstruction approaches.
• Figure 3: Overview of the evaluation process of TokBench.
• Figure 4: Comparison between reconstructed images (right) and original images (left) under different T-ACC and F-Sim metrics. Higher metric values indicate reconstructed images that more closely resemble the original. (Zoom in for better comparison.)
• Figure 5: T-ACC and F-Sim metrics across reconstruction resolutions versus target scales. Smaller scales present greater challenges, and even the best-performing VAE show gap for improvement when compared to the "resize" upper bound.