Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers

TL;DR

This work introduces Visual-Word Tokenizer (VWT), a training-free method to reduce the energy cost of Vision Transformers by dynamically compressing input sequences via two strategies: intra-image dropping of low-variance patches and inter-image grouping of patches into visual words using a Bag-of-Visual-Words. By forming a vocabulary of visual words through k-means clustering on patches and selectively masking or averaging patches with respect to these words, VWT achieves substantial energy savings (up to ~47%) with limited performance loss, and can complement existing compression techniques like quantization. The approach demonstrates improved robustness on certain subgroups and provides rich visualizations of how patches are grouped and dropped, offering a practical path toward efficient online inference in vision systems. The results indicate that selecting intra-image or inter-image strategies depends on the task (global classification vs. detailed captioning), and that VWTs are compatible with other compression pipelines, providing a versatile, training-free tool for energy-aware ViT deployment.

Abstract

The cost of deploying vision transformers increasingly represents a barrier to wider industrial adoption. Existing compression techniques require additional end-to-end fine-tuning or incur a significant drawback to energy efficiency, making them ill-suited for online (real-time) inference, where a prediction is made on any new input as it comes in. We introduce the $\textbf{Visual-Word Tokenizer}$ (VWT), a training-free method for reducing energy costs while retaining performance. The VWT groups visual subwords (image patches) that are frequently used into visual words, while infrequent ones remain intact. To do so, $\textit{intra}$-image or $\textit{inter}$-image statistics are leveraged to identify similar visual concepts for sequence compression. Experimentally, we demonstrate a reduction in energy consumed of up to 47%. Comparative approaches of 8-bit quantization and token merging can lead to significantly increased energy costs (up to 500% or more). Our results indicate that VWTs are well-suited for efficient online inference with a marginal compromise on performance. The experimental code for our paper is also made publicly available.

Figures (9)

• Figure 1: Visualization of the inter-image approach by the Visual-Word Tokenizer (${\mathcal{T}}_{inter}^{100}$ model). Patches that are matched with one another are indicated by identical colors. Higher patch matching is exhibited with the background rather than foreground object across the datasets. Patch matching serves as a rudimentary form of image segmentation by grouping similar non-adjacent visual concepts during inference.
• Figure 2: Overview of the Visual-Word Tokenizer (VWT). An intra-image approach (deploy only). During inference, the pixel variance of the patches is computed with the top-k lowest values being masked. The masked tokens are dropped after positional information is added. An inter-image approach (pre-process & deploy). First, a Bag-of-Visual Words is formed by clustering patches in the pixel space (pre-process). Then, during inference, the minimum pairwise cosine distance between the patches and visual words is computed with values above the threshold being masked (deploy). The unmasked tokens are averaged based on their grouping to the same visual word after positional information is added.
• Figure 3: Visualization of the intra-image approach by the Visual-Word Tokenizer (${\mathcal{T}}_{intra}^{0.5}$ model). Patches with the lowest pixel variance that are dropped are indicated in black. In most cases, the dropped patches correspond to the uniform background which is uninformative. However, in a few cases, the patches of the foreground object are dropped (e.g., airplane in first image for COCO) which is undesirable.
• Figure 4: Visualization of the vocabulary of ${\mathcal{T}}_{inter}^{100}$. Patches from ImageNet-1K are matched to the closest visual word using the Euclidean distance. Visual words are shown to depict basic features such as colors or edges. Each visual word is an average representation of patches that belong to the matched cluster.
• Figure 5: Visualization of long-form captioning on COCO. Longer captions are generated via a length penalty of 2.0 and a maximum length of 40. Interestingly, the smaller vocabulary of ${\mathcal{T}}_{inter}^{100}$ possesses higher descriptiveness and coherence than ${\mathcal{T}}_{intra}^{0.5}$, ${\mathcal{T}}_{inter}^{1000}$, or ${\mathcal{T}}_{inter}^{10000}$ despite its higher compression.