Video Diffusion Transformers are In-Context Learners

TL;DR

Video Diffusion Transformers are In-Context Learners investigates enabling in-context capabilities for video diffusion transformers with minimal tuning. The approach concatenates videos, uses a unified prompt, and applies task-specific LoRA fine-tuning while preserving the original architecture. It demonstrates that long, multi-scene videos (>30 seconds) can be generated with high fidelity and consistent identities, in both reference-free and reference-based settings. The work provides a practical, data-efficient pathway for controllable video generation and releases data, code, and weights to the community.

Abstract

This paper investigates a solution for enabling in-context capabilities of video diffusion transformers, with minimal tuning required for activation. Specifically, we propose a simple pipeline to leverage in-context generation: ($\textbf{i}$) concatenate videos along spacial or time dimension, ($\textbf{ii}$) jointly caption multi-scene video clips from one source, and ($\textbf{iii}$) apply task-specific fine-tuning using carefully curated small datasets. Through a series of diverse controllable tasks, we demonstrate qualitatively that existing advanced text-to-video models can effectively perform in-context generation. Notably, it allows for the creation of consistent multi-scene videos exceeding 30 seconds in duration, without additional computational overhead. Importantly, this method requires no modifications to the original models, results in high-fidelity video outputs that better align with prompt specifications and maintain role consistency. Our framework presents a valuable tool for the research community and offers critical insights for advancing product-level controllable video generation systems. The data, code, and model weights are publicly available at: https://github.com/feizc/Video-In-Context.

Figures (3)

• Figure 1: Examples of in-context generalist for multi-scene video tasks. Four sub-videos are concurrently generated within a single diffusion process that are tuned specifically. A carefully designed prompt template, incorporating distinct scenes, is employed to ensure consistent portrayal and seamless integration of scenes in the generated video sets.
• Figure 2: Examples of in-context generalist for portrait photograph and style transfer tasks. Four sub-videos are generated simultaneously within a single diffusion process being specifically tuned for the desired outcome. Consistent subject identities are preserved across all sub-videos within each set, as demonstrated in the accompanying figure.
• Figure 3: Examples of in-context generalist applied to inpainting and outpainting tasks. A sub-video, enclosed within a red box, is generated based on remaining video clips using a masking operation. This process ensures a consistent style and subjective maintained across all videos within each set.