Video-As-Prompt: Unified Semantic Control for Video Generation

TL;DR

Video-As-Prompt (VAP) introduces a unified semantic-controlled video generation framework that uses reference videos as prompts to enable in-context guidance. It combines a frozen Video Diffusion Transformer with a trainable Mixture-of-Transformers expert, enabling plug-and-play control while avoiding catastrophic forgetting, and introduces a temporally biased Rotary Position Embedding to align prompts temporally. The authors release VAP-Data, the largest semantic-controlled video dataset (~100K paired samples across 100 conditions) to catalyze research and evaluation. Empirically, VAP achieves state-of-the-art performance among open-source methods (38.7% user preference) and exhibits strong zero-shot generalization, rivaling leading commercial models and enabling broad downstream applications. This work advances toward general-purpose, controllable video generation with scalable data and cross-task applicability.

Abstract

Unified, generalizable semantic control in video generation remains a critical open challenge. Existing methods either introduce artifacts by enforcing inappropriate pixel-wise priors from structure-based controls, or rely on non-generalizable, condition-specific finetuning or task-specific architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes this problem as in-context generation. VAP leverages a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture prevents catastrophic forgetting and is guided by a temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. To power this approach and catalyze future research, we built VAP-Data, the largest dataset for semantic-controlled video generation with over 100K paired videos across 100 semantic conditions. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieving a 38.7% user preference rate that rivals leading condition-specific commercial models. VAP's strong zero-shot generalization and support for various downstream applications mark a significant advance toward general-purpose, controllable video generation.

Figures (15)

• Figure 1: Video-As-Prompt (VAP) is a unified semantic-controlled video generation framework: it treats reference videos with wanted semantics as video prompts and controls generation via a plug-and-play, in-context Mixture-of-Transformers expert. Row $1-6$: reference videos used as prompts for diverse semantic-controlled video generation tasks (concept, style, motion, camera). Row $7$: zero-shot generalization results from Video-As-Prompt when given an unseen semantic, demonstrating strong generalizability.
• Figure 2: Controllable Video Generation Paradigms. Structure-Controlled Video Generation (a). The condition is pixel-aligned with the target video. Most works inject conditions (e.g., depth, optical flow, pose) into a DiT via an extra branch using (a) residual addition, leveraging pixel-wise alignment. Semantic-Controlled Video Generation (b, c, d). The condition and target video share the same semantics. Most works use (b) Condition-Specific Overfit or (c) Task-Specific Design: finetuning per semantic or adding task-specific modules. (d) Video-as-Prompt: We use a reference video with the same semantics as prompts and adopt a plug-and-play in-context control framework built on mixture-of-transformers to achieve unified semantic-controlled video generation.
• Figure 3: Overview of Our Proposed VAP-Data. (a) $100$ semantic conditions across $4$ categories: concept, style, camera, and motion; (b) diverse reference images, including animals, humans, objects, and scenes, with multiple variants; and (c) a word cloud of the semantic conditions.
• Figure 4: Overview of Video-As-Prompt. The reference video (with the wanted semantics), target video, and their first frames (reference images) are encoded into latents by the VAE and, together with captions (See top right), form an in-context token sequence $[Ref_{text}, Ref_{video}, Tar_{text}, Tar_{video}]$ (See middle. We omitted term "tokens" for simplicity.). First frame tokens are concatenated with video tokens. We add a temporal bias $\Delta$ to RoPE to avoid nonexistent pixel-aligned priors from the original shared RoPE (See bottom right). The reference video and captions act as the prompts and are fed into a trainable DiT Expert Transformer(See left), which exchanges information bidirectionally with the pre-trained DiT via full attention at each layer, enabling plug-and-play in-context generation.
• Figure 5: Motivation. Ablation visualizations (Semantic: Spin $360^{\circ}$) on structure designs of VAP.