Frequency-Aware Token Reduction for Efficient Vision Transformer

TL;DR

Vision Transformers incur high self-attention costs and suffer from rank collapse and over-smoothing as depth grows. The authors propose a frequency-aware token reduction that splits tokens into high-frequency and low-frequency groups, preserves HF tokens, and aggregates LF content into a DC token, augmented by learnable attention reweighting to prevent collapse. The method includes a practical HF/LF token selection via decomposing attention into low- and high-frequency parts, the use of local DC tokens, and an attention update that integrates DC information, showing consistent accuracy gains and reduced MACs across models and pretrained configurations. This approach yields practical benefits for efficient ViTs and offers insights into why prior token reduction methods may fail from a frequency perspective, with demonstrated applicability to dense prediction as well as classification tasks.

Abstract

Vision Transformers have demonstrated exceptional performance across various computer vision tasks, yet their quadratic computational complexity concerning token length remains a significant challenge. To address this, token reduction methods have been widely explored. However, existing approaches often overlook the frequency characteristics of self-attention, such as rank collapsing and over-smoothing phenomenon. In this paper, we propose a frequency-aware token reduction strategy that improves computational efficiency while preserving performance by mitigating rank collapsing. Our method partitions tokens into high-frequency tokens and low-frequency tokens. high-frequency tokens are selectively preserved, while low-frequency tokens are aggregated into a compact direct current token to retain essential low-frequency components. Through extensive experiments and analysis, we demonstrate that our approach significantly improves accuracy while reducing computational overhead and mitigating rank collapsing and over smoothing. Furthermore, we analyze the previous methods, shedding light on their implicit frequency characteristics and limitations.

Figures (8)

• Figure 1: Frequency analysis of sthe elf-attention layer according to the layer depth. (a) Relative log amplitude of frequency. (b) Relative log amplitude of high-frequency ($1.0 \pi$).
• Figure 2: Analysis of HF tokens and LF tokens. (a) Relative log amplitude of high-frequency ($1.0\pi$). HF tokens contain relatively more high-frequency signals than LF tokens. (b) Similarity with DC-signal. The LF tokens dominantly contain the DC signal of features compared to the HF tokens. (c) Effect of white Gaussian noise (AWGN) on each token set. We report a mean accuracy of 10 trials; the gray dashed line represents the initial accuracy. The results show that the HF tokens are more critical in maintaining the model accuracy compared to the LF tokens.
• Figure 3: Relative log amplitude of high-frequency ($1.0\pi$) of models with (a) different training methods (ViT, DeiT, MAE) and (b) different model size (tiny, small and base).
• Figure 4: (a) Similarity with last layer feature. The gray-dashed line represents the similarity between the models. (b) Similarity with the DC signal feature. In both figures, the gray square represents the reduction layer.
• Figure 5: Results of the ablation study. The gray-dashed line represents the initial accuracy. (a) HF refers to the results of pruning the LF tokens, and LF refers to the results of pruning the HF tokens. (b) Effect of the number of the local window. Each number represents the window size.

Theorems & Definitions (4)

• Proposition 2.1
• Proposition 3.1
• Theorem A.1
• proof