vid-TLDR: Training Free Token merging for Light-weight Video Transformer

TL;DR

vid-TLDR introduces a training-free, two-stage token reduction framework for light-weight video Transformers. It first detects salient regions using an attention-sharpness-based saliency score from early layers, then performs saliency-aware token merging that downweights or removes background tokens. The approach yields significant FLOPs reductions while maintaining competitive performance on video-text retrieval and video QA tasks, and generalizes to other video backbones such as VideoMAE and ViFi-CLIP. This work demonstrates that early-layer token pruning, guided by robust saliency signals, can substantially improve efficiency in video understanding without additional training.

Abstract

Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by the massive number of tokens across the entire video frames, which has been the major barrier to training the model. Further, the patches irrelevant to the main contents, e.g., backgrounds, degrade the generalization performance of models. To tackle these issues, we propose training free token merging for lightweight video Transformer (vid-TLDR) that aims to enhance the efficiency of video Transformers by merging the background tokens without additional training. For vid-TLDR, we introduce a novel approach to capture the salient regions in videos only with the attention map. Further, we introduce the saliency-aware token merging strategy by dropping the background tokens and sharpening the object scores. Our experiments show that vid-TLDR significantly mitigates the computational complexity of video Transformers while achieving competitive performance compared to the base model without vid-TLDR. Code is available at https://github.com/mlvlab/vid-TLDR.

Figures (7)

• Figure 1: Comparison of vid-TLDR (Ours) with UMT li2023unmasked. Without any additional training, vid-TLDR obtains comparable or even better performance than the base model UMT (left) while reducing the considerable computational cost (right). UMT-B (87M) is used.
• Figure 2: Visualization of the attention map of each method in the first layer. Both attentiveness $\bar{a}$ and attention Rollout $\tilde{a}$ confused with foreground objects and the background, also have a temporal bias, resulting in overall low attention in the later frames. These problems are mitigated in our method, focusing on the object across all frames.
• Figure 3: Pipeline of $\textbf{vid-TLDR}$. (a) Given the attention map $A$, the saliency score $s$ is approximated by the $\textit{sharpness function}$$S$ (\ref{['eq:5']}). After that, we generate $\textit{background drop}$ mask $M$ (\ref{['eq:8']}) to minimize the disturbance of background tokens. With $s$ and $M$, we generate $\textit{masked saliency scores}$$\hat{s}$ (\ref{['eq:9']}). (b) Given the tokens $X$ and their corresponding mass $m$, we conduct the matching to group the input tokens. Following that, we update the mass $m$ to $\tilde{m}$ with $\hat{s}$ (\ref{['eq:10']}) to highlight important foreground tokens and minimize the hindrance of background tokens. With updated mass $\tilde{m}$, the grouped tokens are merged into a token $m'$ and $X'$ (\ref{['eq:11']})
• Figure 4: Visualization of attention scores in the first layer of UMT-B li2023unmasked. Given the query $q_i$ denoted as a yellow square, we visualize the attention score $a_i$. Tokens in the foreground objects show a sharper attention map compared to the background tokens.
• Figure 5: The ratio of the sum of the scores in each frame. Given the informativeness scores in the first layer of the video Transformer, we extract the sum of scores in each frame and then calculate the ratio to the total scores across all the frames.