TempoControl: Temporal Attention Guidance for Text-to-Video Models

TL;DR

TempoControl tackles the lack of fine-grained temporal control in text-to-video generation by steering cross-attention at inference time. It introduces three complementary losses—temporal correlation, attention magnitude, and entropy regularization—to shape when and how strongly words grounding in the prompt appear, without retraining the model. The method yields substantial improvements in temporal accuracy across single-, multi-object, and movement tasks while preserving visual quality and diversity, and it extends naturally to audio-aligned generation. Despite higher inference cost and occasional perceptual shifts, TempoControl offers a practical, data-free path to precise temporal grounding and multimodal alignment in diffusion-based video synthesis.

Abstract

Recent advances in generative video models have enabled the creation of high-quality videos based on natural language prompts. However, these models frequently lack fine-grained temporal control, meaning they do not allow users to specify when particular visual elements should appear within a generated sequence. In this work, we introduce TempoControl, a method that allows for temporal alignment of visual concepts during inference, without requiring retraining or additional supervision. TempoControl utilizes cross-attention maps, a key component of text-to-video diffusion models, to guide the timing of concepts through a novel optimization approach. Our method steers attention using three complementary principles: aligning its temporal pattern with a control signal (correlation), adjusting its strength where visibility is required (magnitude), and preserving semantic consistency (entropy). TempoControl provides precise temporal control while maintaining high video quality and diversity. We demonstrate its effectiveness across various applications, including temporal reordering of single and multiple objects, action timing, and audio-aligned video generation. Please see our project page for more details: https://shira-schiber.github.io/TempoControl/.

Figures (16)

• Figure 1: Applications of our inference-time temporal control method (TempoControl), which enables multi-object and single-object control, motion control, and audio alignment. Words that are controlled to appear throughout the entire video are shown in blue. Words that appear according to a temporally varying control signal are shown in orange.
• Figure 2: Motivation for our approach. Top: We show a video generated for the prompt "The video begins with a serene view centered on the cat, with no sign of the dog. In the second half, the dog unexpectedly appears, altering the dynamic of the scene." The top row displays the attention maps for the tokens cat and dog, extracted from the denoising step $t{=}3$, for frames $j{=}2,6,10,14,19$. On the left, the video is generated without our optimization. Despite the prompt specifying that the dog should appear in the second half, it appears early. This behavior is common, as Wan 2.1 often fails to depict objects or movements according to temporal cues in the prompt. On the right, after applying our conditioning method, the dog correctly appears in the second half of the video. Bottom: Temporal attention $a_{i,j}^t$ (blue) versus target mask $m_{i,j}$ (orange), with the corresponding Pearson correlation loss shown.
• Figure 3: Illustration of TempoControl. During a single denoising step $t$, we extract spatial attention maps $\bar{A}^t_{j,i}$ (for word $i$ at temporal index $j$), aggregate to a temporal attention signal $a_{i}^t$, and align it with the target mask vector $m_{i}$ via temporal and spatial losses. Gradients $\nabla \mathcal{L}$ are used to update the latent code $z_t$.
• Figure 4: Entropy regularization helps preserve semantic fidelity.
• Figure 5: Examples generated by Wan 2.1 (left) and our temporally conditioned method (right) for prompts with timing constraints. Prompts are simplified for brevity. Shown are frames from the first, middle, and final thirds of each video.