ACD: Direct Conditional Control for Video Diffusion Models via Attention Supervision

TL;DR

The paper addresses the challenge of achieving direct, reliable conditioning in video diffusion models, where existing guidance strategies often fall short or introduce artifacts.It introduces Attention-Conditional Diffusion (ACD), which supervises the model's attention with sparse 3D-aware object layouts via a dedicated Layout ControlNet and an automated annotation pipeline to ensure scalable, semantically grounded conditioning.Through extensive experiments, ACD demonstrates superior alignment to conditioning inputs while preserving temporal coherence and visual fidelity, outperforming state-of-the-art baselines and maintaining robust camera trajectories.The approach offers a practical pathway to high-fidelity, controllable video synthesis in realistic scenes, with scalable data generation and a focus on attention-level integration rather than output-level conditioning.

Abstract

Controllability is a fundamental requirement in video synthesis, where accurate alignment with conditioning signals is essential. Existing classifier-free guidance methods typically achieve conditioning indirectly by modeling the joint distribution of data and conditions, which often results in limited controllability over the specified conditions. Classifier-based guidance enforces conditions through an external classifier, but the model may exploit this mechanism to raise the classifier score without genuinely satisfying the intended condition, resulting in adversarial artifacts and limited effective controllability. In this paper, we propose Attention-Conditional Diffusion (ACD), a novel framework for direct conditional control in video diffusion models via attention supervision. By aligning the model's attention maps with external control signals, ACD achieves better controllability. To support this, we introduce a sparse 3D-aware object layout as an efficient conditioning signal, along with a dedicated Layout ControlNet and an automated annotation pipeline for scalable layout integration. Extensive experiments on benchmark video generation datasets demonstrate that ACD delivers superior alignment with conditioning inputs while preserving temporal coherence and visual fidelity, establishing an effective paradigm for conditional video synthesis.

Figures (11)

• Figure 1: Visual results generated by the proposed Attention-Conditional Diffusion (ACD) framework. ACD enables direct conditional control in video diffusion models through attention supervision using sparse, 3D-aware object layout signals. Given a single reference image and a sparse object layout with an associated camera trajectory, ACD generates videos that preserve structural semantics and follow the specified camera motion. By applying conditioning at the attention level, the model achieves improved alignment between control inputs and generated content, leading to accurate video synthesis.
• Figure 2: Comparison of conditional control strategies in diffusion models. (a) Classifier-based guidance steers generation using an external classifier, but may produce adversarial artifacts, as the model can increase classifier confidence without genuinely satisfying the intended condition. (b) Classifier-free guidance conditions generation implicitly through joint modeling of data and conditions, yielding strong empirical performance but offering limited fine-grained controllability. (c) Attention-conditional control (ours) applies supervision directly to the model’s attention maps, enabling more direct and semantically grounded alignment between control signals and generated content. "$*$" denotes the generation of the denoised output (commonly represented as $x_0$ or $x_{\mathrm{start}}$), either via direct prediction or by estimating the noise and recovering $x_0$. "$**$" indicates the computation of attention between the original latent representation and a masked (or segmented) latent derived from the control signal.
• Figure 3: Overview of our Attention-Conditional Diffusion (ACD) framework. The input video and its masked version are encoded into visual tokens, while the sparse 3D-aware object layout is converted into layout tokens. These tokens pass through stacked Attention-Conditional DiT blocks, where a router constraint supervises attention maps between masked and unmasked video tokens. Gradients from this constraint update the model parameters. A VAE decoder then reconstructs the video, enabling ACD to generate outputs that closely follow the given layouts and camera trajectories.
• Figure 4: Qualitative comparison of video generation results. Our proposed Attention-Conditional Diffusion (ACD) framework outperforms several state-of-the-art methods, including Stable Virtual Camera, AC3D, and ViewCrafter. The left-most column displays the initial reference images used as input for the generation process. The right-most column shows the ground truth novel views.
• Figure 5: Qualitative comparison of video generation results. Our proposed Attention-Conditional Diffusion (ACD) framework outperforms two variants of Stable Virtual Camera (Seva) on scenes with long camera trajectories. Seva-1 generates videos conditioned on a single reference image, while Seva-4 leverages the first four frames as input.