Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wavelet-Based Image Tokenizer for Vision Transformers

Zhenhai Zhu, Radu Soricut

TL;DR

The paper addresses the inefficiency of the standard patch-convolution image tokenizer in Vision Transformers by introducing a wavelet-based tokenizer. It leverages pixel-space embeddings from a multilevel wavelet transform and a block-sparse projection to produce compact semantic tokens, enabling efficient high-resolution processing. Theoretical analysis clarifies why throughput improves and how the approach naturally resists adversarial perturbations, while experiments on ImageNet show higher top-1 accuracy and increased training throughput with reduced parameter counts. The work suggests promising future directions, including non-uniform token grids and scaling to even higher-resolution imagery in ViT-based models.

Abstract

Non-overlapping patch-wise convolution is the default image tokenizer for all state-of-the-art vision Transformer (ViT) models. Even though many ViT variants have been proposed to improve its efficiency and accuracy, little research on improving the image tokenizer itself has been reported in the literature. In this paper, we propose a new image tokenizer based on wavelet transformation. We show that ViT models with the new tokenizer achieve both higher training throughput and better top-1 precision for the ImageNet validation set. We present a theoretical analysis on why the proposed tokenizer improves the training throughput without any change to ViT model architecture. Our analysis suggests that the new tokenizer can effectively handle high-resolution images and is naturally resistant to adversarial attack. Furthermore, the proposed image tokenizer offers a fresh perspective on important new research directions for ViT-based model design, such as image tokens on a non-uniform grid for image understanding.

Wavelet-Based Image Tokenizer for Vision Transformers

TL;DR

The paper addresses the inefficiency of the standard patch-convolution image tokenizer in Vision Transformers by introducing a wavelet-based tokenizer. It leverages pixel-space embeddings from a multilevel wavelet transform and a block-sparse projection to produce compact semantic tokens, enabling efficient high-resolution processing. Theoretical analysis clarifies why throughput improves and how the approach naturally resists adversarial perturbations, while experiments on ImageNet show higher top-1 accuracy and increased training throughput with reduced parameter counts. The work suggests promising future directions, including non-uniform token grids and scaling to even higher-resolution imagery in ViT-based models.

Abstract

Non-overlapping patch-wise convolution is the default image tokenizer for all state-of-the-art vision Transformer (ViT) models. Even though many ViT variants have been proposed to improve its efficiency and accuracy, little research on improving the image tokenizer itself has been reported in the literature. In this paper, we propose a new image tokenizer based on wavelet transformation. We show that ViT models with the new tokenizer achieve both higher training throughput and better top-1 precision for the ImageNet validation set. We present a theoretical analysis on why the proposed tokenizer improves the training throughput without any change to ViT model architecture. Our analysis suggests that the new tokenizer can effectively handle high-resolution images and is naturally resistant to adversarial attack. Furthermore, the proposed image tokenizer offers a fresh perspective on important new research directions for ViT-based model design, such as image tokens on a non-uniform grid for image understanding.
Paper Structure (21 sections, 20 equations, 1 figure, 3 tables)

This paper contains 21 sections, 20 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Brief Introduction to Wavelet-Based Image Compression
  4. Image Tokenizer
  5. Pixel-Space Token Embedding
  6. Semantic Token Embedding
  7. Proposition 1
  8. Proof:
  9. Proposition 2
  10. Proof:
  11. Operation Count For Transformer Layers
  12. Experimental Results
  13. Setup
  14. Dataset
  15. ViT Models
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Original image and its wavelet coefficients for $YC_bC_r$ channels, where black/white/grey dots correspond to strongly positive / strongly negative / near-to-zero coefficients, respectively.