VideoWeave: A Data-Centric Approach for Efficient Video Understanding

TL;DR

VideoWeave tackles the data scarcity and high compute demands of training long-context video-language models by forming synthetic long-context inputs through splicing short, captioned clips from existing datasets. The method is architecture-agnostic and avoids new annotations or objective changes, reusing original captions and preserving fixed compute. Empirical results on the VideoMME benchmark show that multi-video finetuning via VideoWeave yields higher accuracy than single-video finetuning or image baselines, with random video selection providing a strong baseline and clustering/caption-enrichment offering limited gains. The work highlights a data-centric route to scalable, efficient video-language training and points to future exploration on longer-form videos and broader datasets.

Abstract

Training video-language models is often prohibitively expensive due to the high cost of processing long frame sequences and the limited availability of annotated long videos. We present VideoWeave, a simple yet effective approach to improve data efficiency by constructing synthetic long-context training samples that splice together short, captioned videos from existing datasets. Rather than modifying model architectures or optimization objectives, VideoWeave reorganizes available video-text pairs to expand temporal diversity within fixed compute. We systematically study how different data composition strategies like random versus visually clustered splicing and caption enrichment affect downstream performance on downstream video question answering. Under identical compute constraints, models trained with VideoWeave achieve higher accuracy than conventional video finetuning. Our results highlight that reorganizing training data, rather than altering architectures, may offer a simple and scalable path for training video-language models. We link our code for all experiments here.

Figures (5)

• Figure 1: Our method VideoWeave combines several videos together to form a single input sample. Our method is architecture and task agnostic, and is parametrized by $L$, the number of videos to use for each input pair. In this figure, we visualize $L=2$, representing two distinct video/caption pairs.
• Figure 2: We visualize clusters generated by our modified K-means algorithm. For the experiments described in Section \ref{['sec:clustering']}, we construct input samples by using video clips within a single cluster.
• Figure 3: We show an example caption-caption generation pair where GPT-4o-mini modifies our naive caption into a unified, cohesive video caption.
• Figure 4: We show an example question from VideoMME where our model correctly identifies the correct answer. As is true across all our results, both models are evaluated using an identical set of 16 frames uniformly sampled across the video.
• Figure 5: Per-category performance improvements of multi-video finetuning over (a) single-video finetuning and (b) image baseline on VideoMME.