Koala: Key frame-conditioned long video-LLM

TL;DR

Koala tackles long video question answering by extending short-video vLLMs with sparsely sampled key frames to capture global context and segment-level spatiotemporal reasoning. It introduces two tokenizer modules, Conditioned Segment and Conditioned Video, to fuse global context with local dynamics and inter-segment relations, enabling minutes-long video understanding with a frozen vLLM and lightweight finetuning on HowTo100M. Across EgoSchema, Seed-Bench, and NExT-QA, Koala achieves 3-6% absolute improvements over state-of-the-art baselines and also improves short-term action recognition, demonstrating effective long-range temporal reasoning. The approach offers a practical path to scale vLLMs to longer videos without full end-to-end retraining and indicates strong potential for downstream long-form video tasks.

Abstract

Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key frame-conditioned long video-LLM (Koala), that introduces learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to longer videos. Our approach introduces two new tokenizers that condition on visual tokens computed from sparse video key frames for understanding short and long video moments. We train our proposed approach on HowTo100M and demonstrate its effectiveness on zero-shot long video understanding benchmarks, where it outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.

Figures (10)

• Figure 1: Given a video-Large Language Model that was pretrained on millions of short seconds-long video clips, we propose a lightweight approach (Koala) to extend its short-term video tokenizer function for understanding and answering questions about minutes-long videos. We are the first to use sparsely sampled key frames to condition the LLM. As shown, our Koala approach is more effective at focusing on relevant regions in the input frames than the short vLLMs, allowing it to make more informed predictions based on a more holistic understanding of the video. These regions help facilitate our model in predicting the correct answer to the question (highlighted in green).
• Figure 2: Overview of our full Koala approach. For a given video, we extract a set of coarsely-sampled key frames and non-overlapping frame segments with a much higher sampling rate. We use the key frames to provide high-level global context of the video to compute a final sequence of soft visual tokens that encode both global context as well as fine-grained spatiotemporal information via the Conditioned Segment (CS) and Conditioned Video (CV) tokenizer functions.
• Figure 3: CS and CV tokenizer functions. (a) Our CS tokenizer introduces learnable segment queries and fuses the global semantics of a video with fine-grained frame concept representations within each segment to compute segment tokens. (b) In the CV module, we introduce learnable inter-segment queries as well as temporal concept queries to model the contextual relations between segments.
• Figure 4: Example attention heatmap visualizations on EgoSchema. We provide some qualitative examples of predictions made by our Koala approach and the base Video-Llama model based on what they focus on. We observe that Koala is generally able to focus on relevant regions better than the base vLLM.
• Figure 5: Examples of videos filtered using maximum CLIP scores of video frames with respect to their task labels. We use the CLIP [48] model to compute a similarity score between each extracted frame and the corresponding task label of the video. We generally observe that filtering videos based on the maximum CLIP score of any frame with respect to the task label results in videos with more visual diversity.