HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition

TL;DR

HaltingVT tackles the high computational cost of Joint Space-Time video transformers by applying data-dependent token halting to reduce tokens layer-by-layer. It introduces the Glimpser module for fast shallow-layer pruning and a Motion Loss to emphasize motion cues, all trained end-to-end without extra policy networks. The approach yields strong efficiency-accuracy trade-offs on Mini-Kinetics and ActivityNet, achieving up to 75% top-1 accuracy with relatively low GFLOPs and substantial reductions at very low costs, while preserving or improving accuracy. This work demonstrates that dynamic token reduction, when guided by motion and early pruning, can make high-capacity video transformers practical for large-scale video understanding tasks.

Abstract

Action recognition in videos poses a challenge due to its high computational cost, especially for Joint Space-Time video transformers (Joint VT). Despite their effectiveness, the excessive number of tokens in such architectures significantly limits their efficiency. In this paper, we propose HaltingVT, an efficient video transformer adaptively removing redundant video patch tokens, which is primarily composed of a Joint VT and a Glimpser module. Specifically, HaltingVT applies data-adaptive token reduction at each layer, resulting in a significant reduction in the overall computational cost. Besides, the Glimpser module quickly removes redundant tokens in shallow transformer layers, which may even be misleading for video recognition tasks based on our observations. To further encourage HaltingVT to focus on the key motion-related information in videos, we design an effective Motion Loss during training. HaltingVT acquires video analysis capabilities and token halting compression strategies simultaneously in a unified training process, without requiring additional training procedures or sub-networks. On the Mini-Kinetics dataset, we achieved 75.0% top-1 ACC with 24.2 GFLOPs, as well as 67.2% top-1 ACC with an extremely low 9.9 GFLOPs. The code is available at https://github.com/dun-research/HaltingVT.

Figures (4)

• Figure 1: Illustration of different efficient video analysis strategies. Our HaltingVT (top) reduces overall computation cost by reducing the number of patch tokens, while frame sampling methods (bottom) use a policy network to select key frames, potentially leading to redundant content retention and critical information loss.
• Figure 2: HaltingVT takes the Joint Space-Time video transformer with a token halting mechanism as the core model skeleton. We calculate a halting score for each patch token in each transformer layer, and perform token halting when its cumulative halting score exceeds 1 (red cross in the figure). When the class token reaches the final layer or halts, a weighted sum of class tokens based on their halting scores is transmitted to subsequent classifier for the final prediction. The Glimpser removes redundant tokens in the early stage of inference.
• Figure 3: Comparison of video recognition performance in terms of accuracy and GFLOPs on Mini-Kinetics. HaltingVT achieves the best trade-off between efficiency and effectiveness against SOTAs.
• Figure 4: Visualization for token halting. Each row represents the token halting status of different layers. The patches with lower brightness indicate the tokens that have been halted at that layer.