Don't Look Twice: Faster Video Transformers with Run-Length Tokenization

TL;DR

Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression, which efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length.

Abstract

Transformers are slow to train on videos due to extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods to remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes runs of patches that are repeated over time prior to model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. Our method is content-aware, requiring no tuning for different datasets, and fast, incurring negligible overhead. RLT yields a large speedup in training, reducing the wall-clock time to fine-tune a video transformer by 30% while matching baseline model performance. RLT also works without any training, increasing model throughput by 35% with only 0.1% drop in accuracy. RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80%. Our project page is at https://rccchoudhury.github.io/projects/rlt/.

Figures (6)

• Figure 1: Toy Example. Given a set of input frames, with each square representing a patch, standard tokenization always produces the same number of tokens. RLT compares temporally consecutive patches and removes redundant ones, storing a single token and the run-length instead.
• Figure 2: RLT Overview. RLT works by comparing temporally consecutive patches, and retaining those with L1 difference above a threshold $\tau$. The remaining tokens are augmented with a length encoding to signify their 'run-length' and passed to the transformer.
• Figure 3: Varying Difference Threshold. When comparing the tradeoff between speedup factor and accuracy, RLT is close to baseline performance for low values of $\tau$, with a sharp drop-off after $\tau = 0.1$.
• Figure 4: Sample Visualizations. Tokens that are compressed are visualized in gray. RLT retains tokens that change between frames while removing redundant tokens. In the top example, RLT captures the static background, and in the bottom example, due to camera motion and the motion of the girl, almost no tokens are modified. Video visualizations are available at the project page.
• Figure 5: Effect of $\tau$. With low values of $\tau$, the clearest repeated patches are ablated, but imperceptible variations can prevent some visibly similar tokens from being pruned. Above $\tau = 0.1$, some tokens with slight movement are pruned.