Video-GPT via Next Clip Diffusion

TL;DR

Video-GPT introduces a concise, diffusion-augmented autoregressive framework that treats video clips as language tokens. By using Next Clip Diffusion with Noise-Clean Interleaved Masking, it achieves both short-term generation and long-term prediction within a GPT-like pretraining regime, learned from unlabeled Panda-70M videos. The approach delivers state-of-the-art results on deterministic Physics-IQ forecasting and competitive performance on Kinetics-600, while generalizing to six downstream generation and understanding tasks after fine-tuning. This work demonstrates the potential of self-supervised video learning to acquire broad world knowledge without additional modality supervision, enabling versatile visual intelligence. Future directions include multi-modal pretraining and efficiency improvements to scale further.

Abstract

GPT has shown its remarkable success in natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such details. Motivated by this fact, we propose a concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-art performance on video prediction, which is the key factor towards world modeling (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89). Moreover, it can be well adapted on 6 mainstream video tasks in both video generation and understanding, showing its great generalization capacity in downstream. The project page is at https://zhuangshaobin.github.io/Video-GPT.github.io/.

Figures (12)

• Figure 1: Next clip diffusion. We draw an analogy with GPT's next token prediction and model each video clip as a visual word by denoising the next noisy clip, conditioning on the previous video.
• Figure 2: Video-GPT pretraining framework. The full attention mask is shown in Fig. \ref{['fig:full_mask']}
• Figure 3: Video-GPT inference framework. We iteratively denoise the 2nd noisy clip $\mathbf{NS(2,:)}$ to its clean version $\mathbf{CL(2,:)}$, and use it along with the 1st clean clip $\mathbf{CL(1,:)}$ to condition the prediction of the 3rd noisy clip $\mathbf{NS(3,:)}$. The number of frames in each clip can also vary during inference.
• Figure 4: Fine-tuning Video-GPT on downstream tasks.
• Figure 5: Qualitative results on Physics-IQ Benchmark. The videos predicted by our Video-GPT based on condition frames are more consistent with physical laws than other methods Chen2023SEINESVBai2023SequentialME.