VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generation

TL;DR

VideoUFO introduces a million-scale, user-focused video dataset for text-to-video generation by mining real-world prompts, clustering them into 1,291 topics, and sourcing CC-licensed YouTube videos to produce 1.09M clip captions. It demonstrates that current models underperform on certain user-focused topics and shows a simple model trained on VideoUFO achieves substantial improvements on worst-performing topics via the BenchUFO benchmark. The work also provides ablation analyses on topic curation and clip verification, and releases data and code under CC BY to facilitate broader adoption and further research. The dataset offers a practical path to better align text-to-video models with user needs and real-world use cases.

Abstract

Text-to-video generative models convert textual prompts into dynamic visual content, offering wide-ranging applications in film production, gaming, and education. However, their real-world performance often falls short of user expectations. One key reason is that these models have not been trained on videos related to some topics users want to create. In this paper, we propose VideoUFO, the first Video dataset specifically curated to align with Users' FOcus in real-world scenarios. Beyond this, our VideoUFO also features: (1) minimal (0.29%) overlap with existing video datasets, and (2) videos searched exclusively via YouTube's official API under the Creative Commons license. These two attributes provide future researchers with greater freedom to broaden their training sources. The VideoUFO comprises over 1.09 million video clips, each paired with both a brief and a detailed caption (description). Specifically, through clustering, we first identify 1,291 user-focused topics from the million-scale real text-to-video prompt dataset, VidProM. Then, we use these topics to retrieve videos from YouTube, split the retrieved videos into clips, and generate both brief and detailed captions for each clip. After verifying the clips with specified topics, we are left with about 1.09 million video clips. Our experiments reveal that (1) current 16 text-to-video models do not achieve consistent performance across all user-focused topics; and (2) a simple model trained on VideoUFO outperforms others on worst-performing topics. The dataset and code are publicly available at https://huggingface.co/datasets/WenhaoWang/VideoUFO and https://github.com/WangWenhao0716/BenchUFO under the CC BY 4.0 License.

Figures (11)

• Figure 1: VideoUFO is the first dataset curated in alignment with real-world users’ focused topics for text-to-video generation. Specifically, the dataset comprises over $1.09$ million video clips spanning $1,291$ topics. Here, we select the top $20$ most popular topics for illustration. Researchers can use our VideoUFO to train or fine-tune their text-to-video generative models to better meet users’ needs.
• Figure 2: The glowing firefly: (a) generated by Sora openai2024sora and (b) captured in a real video. The generated firefly is noticeably different from its real-life counterpart and thus unsatisfying. We attribute this primarily to a lack of exposure to such topics.
• Figure 3: Each data point in our VideoUFO includes a video clip, an ID, a topic, start and end times, a brief caption, and a detailed caption. Beyond that, we evaluate each clip with six different video quality scores from VBench huang2024vbench.
• Figure 4: The semantic distribution of users’ focused topics. It is visualized by WizMapwang2023wizmap. Please zoom in or visit https://poloclub.github.io/wizmap/?dataURL=https://huggingface.co/datasets/WenhaoWang/Public/resolve/main/data_vidufo.ndjson&gridURL=https://huggingface.co/datasets/WenhaoWang/Public/resolve/main/grid_vidufo.json to see the details.
• Figure 5: The number of user-focused topics covered by recent video datasets. None successfully includes all user-focused topics.