SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection
Foozhan Ataiefard, Walid Ahmed, Habib Hajimolahoseini, Saina Asani, Farnoosh Javadi, Mohammad Hassanpour, Omar Mohamed Awad, Austin Wen, Kangling Liu, Yang Liu
TL;DR
SkipViT tackles the high computational cost of Vision Transformers by dropping less informative image patches using CLS-attention-derived scores while preserving accuracy through a residual skip path that reincorporates dropped tokens in later layers. The method requires no additional parameters and achieves substantial training throughput gains (e.g., up to ~16% faster) with negligible Top-1 accuracy loss on ViT-small trained from scratch on ImageNet1K. By carefully selecting which tokens to drop and when to reintroduce them, SkipViT demonstrates stable performance improvements with a clear trade-off between speed and accuracy. The approach is validated on ViT-small, with findings suggesting potential scalability to larger ViT models and datasets, though current experiments are limited to the smaller architecture and dataset.
Abstract
Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A.