Principles of Visual Tokens for Efficient Video Understanding

TL;DR

This work addresses the computational cost of video transformers by examining the value of visual tokens and revealing that many token-reduction methods underperform simple random dropping. It introduces five principles about token value, notably a Pareto distribution, and leverages these insights to propose LITE, a lightweight, oracle-guided token selector inserted before the transformer. LITE uses a gradient-based oracle to train a small MLP that imitates token importance, enabling adaptive budgeting that reduces compute while preserving accuracy, and it exhibits strong performance on Kinetics-400 and Something-Something-V2 with zero-shot transfer to new datasets and tasks (e.g., AVA). The paper demonstrates that token efficiency can be achieved without retraining for new domains, offering practical gains for deploying efficient video understanding models and guiding future research on visual tokens.

Abstract

Video understanding has made huge strides in recent years, relying largely on the power of transformers. As this architecture is notoriously expensive and video data is highly redundant, research into improving efficiency has become particularly relevant. Some creative solutions include token selection and merging. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the baseline of randomly discarding tokens. In this paper we take a closer look at this phenomenon and observe 5 principles of the nature of visual tokens. For example, we observe that the value of tokens follows a clear Pareto-distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. We build on these and further insights to propose a lightweight video model, LITE, that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics-400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy. Experiments also show that LITE generalizes across datasets and even other tasks without the need for retraining.

Figures (7)

• Figure 1: Left: Given an input video (top row), the proposed LITE method is able to select a subset (non-white patches) of discriminative tokens for any given computational budget. Right: LITE is lightweight and efficient, achieving the optimal Pareto-front over state-of-the-art in the trade-off between computational cost (GFLOPs) and accuracy. The plot shows the results on Something-Something-V2.
• Figure 2: Random sampling of tokens is comparable to or outperforms most sophisticated models for token selection, as well as other sampling baselines. The "P-Ratio" represents the proportion of tokens selected relative to the total number of tokens per video.
• Figure 3: Histogram of value of tokens as predicted by the oracle, showing a very clear Pareto-like distribution. Easy classes have thinner tails than hard classes.
• Figure 4: Sample heatmaps for different visual cues. From top to bottom: input video frames, oracle, magnitude of motion vectors, saliency, and attention.
• Figure 5: Impact of dropping 70% of tokens in easy classes vs hard classes. The X-axis is the accuracy when using all tokens. The Y-axis is the decay of the accuracy when dropping 70% of tokens, as a percentage. Easier classes suffer less when the computational budget is reduced.