ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Access

TL;DR

ViMix-14M tackles the core data bottleneck in open-source video-language research by delivering a large, crawl-free video-text dataset with long-form captions across seven sources. The dataset provides three caption granularities generated via Qwen2.5-VL-Instruct-7B, guided by ground-truth labels, and evaluated with VBench for multi-dimensional video quality. Empirical results show consistent gains in multimodal retrieval, text-to-video generation quality, and VQA performance compared to baselines. This resource lowers reproducibility barriers and offers a scalable, diverse benchmark to advance open-source video foundation-model development and analysis of caption quality on downstream tasks.

Abstract

Text-to-video generation has surged in interest since Sora, yet open-source models still face a data bottleneck: there is no large, high-quality, easily obtainable video-text corpus. Existing public datasets typically require manual YouTube crawling, which yields low usable volume due to link rot and access limits, and raises licensing uncertainty. This work addresses this challenge by introducing ViMix-14M, a curated multi-source video-text dataset of around 14 million pairs that provides crawl-free, download-ready access and long-form, high-quality captions tightly aligned to video. ViMix-14M is built by merging diverse open video sources, followed by unified de-duplication and quality filtering, and a multi-granularity, ground-truth-guided re-captioning pipeline that refines descriptions to better match actions, scenes, and temporal structure. We evaluate the dataset by multimodal retrieval, text-to-video generation, and video question answering tasks, observing consistent improvements over counterpart datasets. We hope this work can help removing the key barrier to training and fine-tuning open-source video foundation models, and provide insights of building high-quality and generalizable video-text datasets.

Figures (8)

• Figure 1: Multi-granularity caption generation and filtering pipeline. Given video frames and ground truth captions (when available), we employ Qwen2.5-VL bai2025qwen2 with three prompt levels to generate captions at varying granularities: short captions for brief content summarization, middle captions capturing scene details, and long captions encoding spatial relations and reasoning. Ground truth labels serve as optional contextual guidance to enhance caption quality but are not required for caption generation. VBench huang2023vbench evaluates video quality across five dimensions: subject consistency, temporal flickering, background consistency, imaging quality, and aesthetic quality. The fishing scene example demonstrates how caption granularity progressively captures richer visual and contextual details.
• Figure 2: Caption and video statistics. Left: frequency of captions (y-axis) by word count (x-axis) for short, middle, and long granularities. Right: percentage distribution of video durations.
• Figure 3: Multi-dimensional caption quality comparison. Our captions demonstrate superior semantic richness across all evaluated dimensions compared to existing datasets.
• Figure 4: Video generation quality comparison across caption granularities. Origin Caption (e.g. InternVid wang2023internvid) provides the basic scene; Short Caption adds hand-mushroom interaction; Middle Caption correctly predicts the gloved hand with color; Long Caption achieves the best results with improved hand posture and correct contact direction. Quality hierarchy: Origin $<$ Short $<$ Middle $<$ Long.
• Figure 5: Caption evaluation via video question-answering tasks. Comparison between baseline captions (a) and ours (b) across multiple datasets. Color markers show answer coverage. Our captions capture richer visual details, leading to more accurate responses for attribute-specific and content-focused questions.