Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformers

TL;DR

This work tackles the computational burden of Vision Transformers by preserving spatial locality during token reduction. It introduces Hilbert-curve reordering combined with neighbor-aware pruning (NAP) and adjacent-token merging (MAT), plus a hybrid HyNAP that combines pruning and merging in a DiffRate-based framework. Empirical results show strong accuracy–efficiency trade-offs across ViT variants, with NAP improving throughput at modest accuracy loss and MAT excelling at lower merging ratios, while HyNAP achieves substantial FLOP reductions and throughput gains with minimal accuracy loss. The findings highlight the importance of spatial continuity and local context for efficient ViT design, offering practical, plug-and-play reductions that can be integrated into existing ViT pipelines.

Abstract

Vision Transformers (ViTs) have achieved remarkable success in visual recognition tasks, but redundant token representations limit their computational efficiency. Existing token merging and pruning strategies often overlook spatial continuity and neighbor relationships, resulting in the loss of local context. This paper proposes novel neighbor-aware token reduction methods based on Hilbert curve reordering, which explicitly preserves the neighbor structure in a 2D space using 1D sequential representations. Our method introduces two key strategies: Neighbor-Aware Pruning (NAP) for selective token retention and Merging by Adjacent Token similarity (MAT) for local token aggregation. Experiments demonstrate that our approach achieves state-of-the-art accuracy-efficiency trade-offs compared to existing methods. This work highlights the importance of spatial continuity and neighbor structure, offering new insights for the architectural optimization of ViTs.

Figures (4)

• Figure 1: Token order. Illustration of token sequence reordering from row-major order (left) to Hilbert curve order (right), which better preserves spatial locality in the 1D token sequence. The reordered tokens are flattened below, demonstrating how Hilbert-curve ordering brings adjacent image patches closer in the 1D sequence.
• Figure 2: Pipeline of Hilbert Curve–based Neighbor‑Aware Token Reduction. After the image is patched, it is reordered by the Hilbert curve and then sent to the ViT block. In each ViT block, a token reduction module is inserted between the attention and MLP modules to reduce the number of tokens. The specific reduction method can be token pruning (NAP), token merging (NAP), or a combination of the two. After the token reduction module, the computational burden of subsequent modules and blocks is reduced.
• Figure 3: Comparison of NAP, MAT, and other token reduction methods at different token reduction rates
• Figure 4: Performance under different ordering schemes.