LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoders

TL;DR

LV-MAE addresses the challenge of long-video understanding by decoupling short-video feature extraction from long-range dependency modeling. It learns long-range representations via a masked-embedding autoencoder that operates on embeddings produced by off-the-shelf multimodal encoders, enabling efficient pre-training on hours-long videos. The method achieves state-of-the-art results on LVU, COIN, and Breakfast with minimal downstream fine-tuning, and provides interpretable reconstructions through caption-based retrieval in the short-video embedding space. This approach offers scalable long-form video understanding and opens avenues for retrieval and generation tasks, while depending on the quality of frozen short-video embeddings.

Abstract

In this work, we introduce long-video masked-embedding autoencoders (LV-MAE), a self-supervised learning framework for long video representation. Our approach treats short- and long-span dependencies as two separate tasks. Such decoupling allows for a more intuitive video processing where short-span spatiotemporal primitives are first encoded and are then used to capture long-range dependencies across consecutive video segments. To achieve this, we leverage advanced off-the-shelf multimodal encoders to extract representations from short segments within the long video, followed by pre-training a masked-embedding autoencoder capturing high-level interactions across segments. LV-MAE is highly efficient to train and enables the processing of much longer videos by alleviating the constraint on the number of input frames. Furthermore, unlike existing methods that typically pre-train on short-video datasets, our approach offers self-supervised pre-training using long video samples (e.g., 20+ minutes video clips) at scale. Using LV-MAE representations, we achieve state-of-the-art results on three long-video benchmarks -- LVU, COIN, and Breakfast -- employing only a simple classification head for either attentive or linear probing. Finally, to assess LV-MAE pre-training and visualize its reconstruction quality, we leverage the video-language aligned space of short video representations to monitor LV-MAE through video-text retrieval. Code is available at https://github.com/amazon-science/lv-mae.

Figures (6)

• Figure 1: Overview of the LV-MAE method. LV-MAE first utilizes short-video representations extracted by advanced multimodal off-the-shelf encoders (e.g., LanguageBind zhu2024languagebind, InternVideo2 Wang2024InternVideo2SV) to capture low-level knowledge of atomic actions and localized events. Next, we pre-train a masked embedding autoencoder to learn long-range dependencies across video segments in a self-supervised manner. After pre-training, the LV-MAE encoder is used to extract high-level representations for long-video downstream tasks.
• Figure 2: Masking ratio. The y-axes are LVU average accuracy scores. A moderate masking ratio of $40-50\%$ works well for attentive probing.
• Figure 3: Interpretable predictions -- examples: Each row visualizes three consecutive five-second segments. Above each segment, we show the original caption for the visible tokens and the retrieved caption for the reconstructed masked tokens. As shown, the model successfully reconstructs the semantic meaning of the masked embeddings, offering insight into the model's effectiveness and capabilities.
• Figure 4: Interpretable predictions -- scheme: For each reconstructed masked embedding, we perform retrieval against a large set of captions collected from the MovieClip dataset. The top matches can then be used to assess the model's quality.
• Figure 5: Average performance of LVU benchmark rises monotonically as the frame count increases.