TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment

TL;DR

Text-Only Pre-Alignment (TOPA) is introduced, a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data, and proves competitive with recent GPT-3.5-based video agents.

Abstract

Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains a challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Only Pre-Alignment (TOPA), a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data. Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text data. Then, these annotated textual videos are used to pre-align a language-only LLM with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor to align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is an effective and efficient framework for aligning video content with LLMs. In particular, without training on any video data, the TOPA-Llama2-13B model achieves a Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, Egoschema. This performance surpasses previous video-text pre-training approaches and proves competitive with recent GPT-3.5-based video agents.

Figures (18)

• Figure 1: Overview of the proposed Text-Only Pre-Alignment (TOPA) framework. Left: The pipeline used for generating the TextVid dataset. Right: The video-LLM alignment framework. During text-only pre-alignment, the LLM learns to process continuous CLIP text features. In zero-shot inference, the LLM uses projected CLIP visual features as input. Additionally, TOPA supports supervised fine-tuning on downstream video datasets to further improve the performance.
• Figure 2: Examples of TOPA-LLama2-13B for video-language understanding. Given a video, TOPA is able to summarize the video content and answer the questions.
• Figure 3: Results of finetuning TOPA with various ratios of training data.
• Figure 4: Qualitative results on NeXT-QA. TOPA effectively performs complex video understanding tasks. Additionally, a failure case is also shown in the figure, i.e., in the last sample, TOPA failed to accurately count the number of people.
• Figure 5: EgoSchema presents unique challenges compared to previous video benchmarks. The questions in EgoSchema are complex and demand advanced video capabilities, encompassing both recognition and reasoning skills.