DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

TL;DR

<3-5 sentence high-level summary> DiTCtrl tackles the challenge of zero-shot, multi-prompt longer video generation by analyzing and leveraging the attention mechanisms of MM-DiT to enable precise, mask-guided semantic control across sequential prompts. It introduces a KV-sharing strategy to maintain content consistency and a latent blending mechanism to produce smooth transitions between prompts, all without additional training. The authors also present MPVBench, a dedicated benchmark with metrics to evaluate multi-prompt transitions, and demonstrate state-of-the-art performance with efficient inference. This work paves the way for coherent long-form video generation and editing within diffusion-transformer frameworks, with practical implications for flexible, prompt-driven video creation in real-world scenarios.

Abstract

Sora-like video generation models have achieved remarkable progress with a Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current video generation models predominantly focus on single-prompt, struggling to generate coherent scenes with multiple sequential prompts that better reflect real-world dynamic scenarios. While some pioneering works have explored multi-prompt video generation, they face significant challenges including strict training data requirements, weak prompt following, and unnatural transitions. To address these problems, we propose DiTCtrl, a training-free multi-prompt video generation method under MM-DiT architectures for the first time. Our key idea is to take the multi-prompt video generation task as temporal video editing with smooth transitions. To achieve this goal, we first analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models, enabling mask-guided precise semantic control across different prompts with attention sharing for multi-prompt video generation. Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts without additional training. Besides, we also present MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation. Extensive experiments demonstrate that our method achieves state-of-the-art performance without additional training.

Figures (23)

• Figure 1: Our method DiTCtrl takes multiple text prompts as input and demonstrates superior capability in generating longer videos with multiple events, long-range coherence and smooth transitions as output.
• Figure 2: MM-DiT Attention Analysis. We find the attention matrix in MM-DiT attention can be divided into four different regions. As for the prompt of " a cat watch a black mouse", each text token shows a high-light response using the average of the text-to-video and video-to-text attention.
• Figure 3: MM-DiT Text-to-Text and Video-to-Video Attention Visualization. We find that the current MM-DiT has a stronger potential to construct the individual attention in the previous UNet-like structure chen2023videocrafter1chen2024videocrafter2wang2023modelscope.
• Figure 4: Pipeline of the proposed DiTCtrl. Note that initial latents are assumed to be 5 frames here. The first three frames are used to generate the contents of $P_{i-1}$, and the last three frames are used to generate contents of $P_i$. The pink latent represents the overlapping frame, while the blue and green latents are used to distinguish different prompt segments. Our method tries to synthesize content-consistent videos based on multi-prompts. The first video is synthesized with source text prompt $P_{i-1}$. During the denoising process for video synthesis, we convert the full-attention into masked-guided KV-sharing strategy to query video contents from source video $\mathcal{V}_{i-1}$, so that we can synthesize content-consistent video under the modified target prompt $P_i$.
• Figure 5: Latent blending strategy for video transition between video clips.