Video, How Do Your Tokens Merge?

TL;DR

This work tackles the prohibitive compute cost of video transformers by introducing training-free video token merging that progressively reduces token counts during inference. It extends image-based token merging to video via two schemes—Spatio-Temporal and Divided Space-Time—merging tokens based on token-key similarity and weighting by multiplicity, with proportional attention to stabilize subsequent layers. Across four models and three action datasets, the method yields around 2.5x throughput with modest accuracy loss, and is particularly robust for spatio-temporal models like ViViT and VideoMAE, while divided space-time variants suffer on fine-grained temporal tasks. Qualitative analyses indicate merging is largely visual rather than semantic, and layer choice governs the trade-off between speed and accuracy, offering a practical drop-in acceleration for video understanding systems.

Abstract

Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around $2.5$X while maintaining accuracy (avg. $-0.55\%$ for ViViT). Code available at https://github.com/sjpollard/video-how-do-your-tokens-merge.

Figures (20)

• Figure 1: Video Token Merging reduces computation of video transformer models by successively merging tokens without re-training or additional learned parameters. We show how an input video has its tokens merged across different layers.
• Figure 2: The merging process first separates tokens into two sets. Similarities are calculated and a one-to-many bipartite matching between tokens in each set is found. Finally, the top $r$ edges are kept and these are merged based on the similarity between tokens.
• Figure 3: (Left) curves corresponds to accuracy with ViViT and VideoMAE on K$400$ when increasing $r$ (the number of tokens merged) up to its limit. The $x$-axis is the percentage (relative to the original total) of tokens dropped per layer. (Right) figure displays the accuracy against speedup gained for these $r$ values.
• Figure 4: Impact on confusion matrices from Token Merging a VideoMAE model. The first $10$ verb and noun classes are displayed left and right respectively from VideoMAE on EK-$100$. Red indicates less predictions and blue indicates more predictions.
• Figure 5: Visualisation of the final merged tokens for an SSv$2$ clip of "covering salt shaker with a towel", produced with VideoMAE. Tokens $1$ and $2$ capture the white salt shaker. The model struggles more with the blue towel, with it splitting into tokens $3$ -- $8$.