ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models

TL;DR

ShotAdapter addresses the challenge of generating multi-shot videos with discrete cuts while preserving a coherent character identity and adaptable backgrounds. It fine-tunes a pre-trained text-to-video diffusion model using a learnable transition token and a localized attention masking scheme, enabling shot-level control via shot-specific prompts. A novel data collection and post-processing pipeline constructs multi-shot datasets from single-shot videos to train the model with minimal fine-tuning (about 5000 iterations). Extensive experiments against baselines and a user study show improvements in identity and background consistency and competitive text alignment, with demonstrated generalization to 2–8 shots. This framework paves the way for controllable multi-shot video generation suitable for film and storytelling applications.

Abstract

Current diffusion-based text-to-video methods are limited to producing short video clips of a single shot and lack the capability to generate multi-shot videos with discrete transitions where the same character performs distinct activities across the same or different backgrounds. To address this limitation we propose a framework that includes a dataset collection pipeline and architectural extensions to video diffusion models to enable text-to-multi-shot video generation. Our approach enables generation of multi-shot videos as a single video with full attention across all frames of all shots, ensuring character and background consistency, and allows users to control the number, duration, and content of shots through shot-specific conditioning. This is achieved by incorporating a transition token into the text-to-video model to control at which frames a new shot begins and a local attention masking strategy which controls the transition token's effect and allows shot-specific prompting. To obtain training data we propose a novel data collection pipeline to construct a multi-shot video dataset from existing single-shot video datasets. Extensive experiments demonstrate that fine-tuning a pre-trained text-to-video model for a few thousand iterations is enough for the model to subsequently be able to generate multi-shot videos with shot-specific control, outperforming the baselines. You can find more details in https://shotadapter.github.io/

Figures (8)

• Figure 1: ShotAdapter is a lightweight framework that enables text-to-multi-shot video generation by fine-tuning a pre-trained text-to-video model. It allows control over number and duration of shots as well as shot content through shot-specific text prompts. The framework maintains character identity while being able to preserve backgrounds (e.g.3rd row) or transition to new ones (e.g.4th row), featuring distinct activities (e.g. playing guitar, then using laptop) and perspectives.
• Figure 2: Comparison with baselines. Each row displays frames from generated 2-shot videos, guided by shot-specific text prompts. The left column shows results with same background and different activities, while the right column presents results both with different backgrounds and activities.
• Figure 3: Fine-tuning framework with transition token and local attention masking. (a) ShotAdapter fine-tunes a pre-trained T2V model by incorporating "transition tokens" (highlighted in light blue). We use $n-1$ transition tokens, initialized as learnable parameters, alongside an $n$-shot video with shot-specific prompts, which are fed through the pre-trained T2V model. (b) The model processes the concatenated input token sequence, guided by a "local attention mask" through joint attention layers within DiT blocks. (c) The local attention mask is structured to ensure that transition tokens interact only with the visual frames where transitions occur, while each textual token interacts exclusively with its corresponding visual tokens.
• Figure 4: Multi-shot video dataset collection pipeline. A high-level overview of this pipeline is presented in (a). Our first method (gray box in (b)) samples videos with large motion, randomly splits them into $n$-shots with varied durations, and concatenates them into multi-shot videos. Our second method (yellow box in (b)) randomly samples $n$ videos from pre-clustered groups containing videos of the same identities and concatenates them to form a multi-shot video. Finally, we post-process (c) the multi-shot videos to ensure identity consistency and obtain shot-specific captions using LLaVA-NeXT zhang2024llavanextvideo.
• Figure 5: Qualitative results. Our approach enables multi-shot video generation depicting different actions and background guided by shot-specific prompts. In the 2nd row, the shots maintain a consistent background while capturing different perspectives, whereas the 3rd row depicts the same woman in related backgrounds that subtly change in response to the prompt. For complete videos, see Supplementary.