EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens

TL;DR

EVEREST tackles the heavy computational and memory demands of Masked Video Autoencoders by introducing redundancy-robust token generation and information-intensive frame selection. It selectively preserves tokens that exhibit large temporal changes and adaptively samples frames with dense information, enabling training on a single node with eight GPUs while achieving competitive performance on standard benchmarks and Ego4D OSCC. The method delivers substantial efficiency gains, reducing pre-training and fine-tuning GFLOPs and memory usage, thereby lowering the barrier to advanced video representation learning. This work advances eco-friendly, accessible VRL by focusing computation on informative spatiotemporal regions and frames.

Abstract

Masked Video Autoencoder (MVA) approaches have demonstrated their potential by significantly outperforming previous video representation learning methods. However, they waste an excessive amount of computations and memory in predicting uninformative tokens/frames due to random masking strategies. (e.g., over 16 nodes with 128 NVIDIA A100 GPUs). To resolve this issue, we exploit the unequal information density among the patches in videos and propose EVEREST, a surprisingly efficient MVA approach for video representation learning that finds tokens containing rich motion features and discards uninformative ones during both pre-training and fine-tuning. We further present an information-intensive frame selection strategy that allows the model to focus on informative and causal frames with minimal redundancy. Our method significantly reduces the computation and memory requirements of MVA, enabling the pre-training and fine-tuning on a single machine with 8 GPUs while achieving comparable performance to computation- and memory-heavy baselines on multiple benchmarks and the uncurated Ego4D dataset. We hope that our work contributes to reducing the barrier to further research on video understanding.

Figures (10)

• Figure 1: Efficiency of our EVEREST against VideoMAE on K400 dataset.(a) GFLOPs for pre-training and fine-tuning. The bubble size is proportional to the GFLOPs of the model. (b) Memory consumption using one node equipped with 8$\times$ A100 (80GB). VideoMAE with ViT-B and -L fails to deploy the model due to out-of-memory if the batch sizes are 512 or larger. For a ViT-L backbone with a batch size of 256, our method achieves about $4 \times$ less memory consumption than VideoMAE. Please see \ref{['tab:vs_videomae', 'tab:mem-allocation2']} for detailed results.
• Figure 2: Overview of EVEREST. Our redundancy-robust mask generator selects tokens with a large disparity with the paired ones in the previous time dimension, indicating that they include rich motion features. Then, the model focuses on learning representation by reconstructing only sparsified videos containing abundant spatiotemporal information, which makes the VRL surprisingly efficient.
• Figure 3: Information-intensive frame selection. We adaptively select frames based on the ReRo token frequency, which indicates significance compared to frames.
• Figure 4: Visualization of the proposed information-intensive frame selection on an uncurated dataset, Ego4D. Unlike prior works that uniformly samples frames similar to each other, we adaptively sample the given video (24 frames) by probabilistic sampling the frames that have distinct spatiotemporal features (non-blurred frames).
• Figure 5: Performance & GFLOPs Comparison on UCF101 dataset. (Left) EVEREST outperforms VideoMAE for both masking ratios (75% and 90%) even at significantly fewer training epochs. (Right) Our EVEREST reduces GFLOPs during pre-training and fine-tuning compared to VideoMAE and ST-MAE.