Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks

TL;DR

This work addresses the shortage of large-scale Chinese video-language data by releasing Youku-mPLUG, a 10M video-text corpus with safety, diversity, and quality controls, plus a 0.3M benchmark suite across video category classification, captioning, and retrieval. It introduces mPLUG-video, a decoder-only VLP model that leverages a frozen LLM, a TimeSformer-based video encoder, and a visual abstractor to generate multimodal representations, achieving state-of-the-art results on the proposed benchmarks (e.g., CIDEr 68.9, Top-1 category accuracy 80.57%). The dataset and models are also extended to a Bloomz-based Chinese multimodal LLM with 1.7% trainable parameters, demonstrating strong zero-shot instruction and video understanding capabilities. Overall, Youku-mPLUG enables rigorous evaluation and advances in Chinese video-language modeling, with practical impact for multimodal AI in Chinese contexts.

Abstract

To promote the development of Vision-Language Pre-training (VLP) and multimodal Large Language Model (LLM) in the Chinese community, we firstly release the largest public Chinese high-quality video-language dataset named Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing website, with strict criteria of safety, diversity, and quality. Youku-mPLUG contains 10 million Chinese video-text pairs filtered from 400 million raw videos across a wide range of 45 diverse categories for large-scale pre-training. In addition, to facilitate a comprehensive evaluation of video-language models, we carefully build the largest human-annotated Chinese benchmarks covering three popular video-language tasks of cross-modal retrieval, video captioning, and video category classification. Youku-mPLUG can enable researchers to conduct more in-depth multimodal research and develop better applications in the future. Furthermore, we release popular video-language pre-training models, ALPRO and mPLUG-2, and our proposed modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG. Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1% improvement in video category classification. Besides, mPLUG-video achieves a new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in video category classification and 68.9 CIDEr score in video captioning, respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate impressive instruction and video understanding ability. The zero-shot instruction understanding experiment indicates that pretraining with Youku-mPLUG can enhance the ability to comprehend overall and detailed visual semantics, recognize scene text, and leverage open-domain knowledge.

Figures (9)

• Figure 1: Random sampled examples in Youku-mPLUG.
• Figure 2: The distribution of the number of videos in each common category.
• Figure 3: Youku-mPLUG dataset statistics: we report the histogram of video duration in seconds (left), the histogram of title length in words (middle), and the ratios of the categories in each super-category (right).
• Figure 4: The overview of mPLUG-video.
• Figure 5: Human evaluation about zero-shot video instruction understanding on 65 cases.