Focal Guidance: Unlocking Controllability from Semantic-Weak Layers in Video Diffusion Models

TL;DR

This work identifies Semantic-Weak Layers in Diffusion Transformer–based Image-to-Video models as a consequence of Condition Isolation, which weakens text guidance during denoising. It introduces Focal Guidance (FG), combining Fine-grained Semantic Guidance (FSG) and Attention Cache (AC) to restore fine-grained text–visual grounding and propagate semantic signals to weak layers. The authors also present a benchmark for evaluating instruction-following in I2V and demonstrate notable improvements on Wan2.1-I2V and HunyuanVideo-I2V, highlighting FG’s potential for improving controllability with minimal post-training. The approach offers a practical, model-agnostic pathway to align textual instructions with reference frames in video generation, with implications for more reliable semantic control in diffusion-based video synthesis.

Abstract

The task of Image-to-Video (I2V) generation aims to synthesize a video from a reference image and a text prompt. This requires diffusion models to reconcile high-frequency visual constraints and low-frequency textual guidance during the denoising process. However, while existing I2V models prioritize visual consistency, how to effectively couple this dual guidance to ensure strong adherence to the text prompt remains underexplored. In this work, we observe that in Diffusion Transformer (DiT)-based I2V models, certain intermediate layers exhibit weak semantic responses (termed Semantic-Weak Layers), as indicated by a measurable drop in text-visual similarity. We attribute this to a phenomenon called Condition Isolation, where attention to visual features becomes partially detached from text guidance and overly relies on learned visual priors. To address this, we propose Focal Guidance (FG), which enhances the controllability from Semantic-Weak Layers. FG comprises two mechanisms: (1) Fine-grained Semantic Guidance (FSG) leverages CLIP to identify key regions in the reference frame and uses them as anchors to guide Semantic-Weak Layers. (2) Attention Cache transfers attention maps from semantically responsive layers to Semantic-Weak Layers, injecting explicit semantic signals and alleviating their over-reliance on the model's learned visual priors, thereby enhancing adherence to textual instructions. To further validate our approach and address the lack of evaluation in this direction, we introduce a benchmark for assessing instruction following in I2V models. On this benchmark, Focal Guidance proves its effectiveness and generalizability, raising the total score on Wan2.1-I2V to 0.7250 (+3.97\%) and boosting the MMDiT-based HunyuanVideo-I2V to 0.5571 (+7.44\%).

Figures (8)

• Figure 1: Visualization of semantic alignment within the Wan2.1-I2V wan2025wan, quantified by the cosine similarity between visual features and keyword textual features. The features are sampled at evenly spaced inference steps and network layers. The heatmap reveals that the initial and final layers exhibit stronger and more accurate alignment with the target words, while several intermediate layers show noticeably degraded and noisy responses.
• Figure .1: Illustrative qualitative examples generated by FG-Wan2.1-I2V 14B across three dimensions: human motion, human interaction, and dynamic attribute changes. These cases demonstrate the model’s ability to produce realistic, temporally consistent, and semantically coherent video outputs under diverse scenarios.
• Figure 2: Statistical analysis of visual-textual similarity across 50 samples. We evaluate the semantic responsiveness of DiT layers by measuring Moran’s I ( \ref{['fig:MoranI']}) and standard deviation ( \ref{['fig:Std']}) of normalized visual-textual similarity maps. Consistent with the results in Fig. \ref{['fig:heatmap']}, the initial and final layers show stronger and more stable responses to textual keywords, while intermediate layers exhibit weakened semantic alignment.
• Figure 3: Overview of the Focal Guidance framework. FG consists of two main components: Fine-grained Semantic Guidance and Attention Cache. (a) Fine-grained Semantic Guidance enhances the accuracy of information conditioning and reduces the model's learning complexity by coupling the fine-grained relationships among the VAE-encoded reference frame, image encoder features, and text conditions. (b) Attention Cache leverages the semantic-responsive layers' attention patterns to guide the injection of conditions into layers with weak semantic responses.
• Figure 4: Qualitative comparison of controllability across mainstream open-source I2V models. Existing methods often fail to reliably ground the text instruction in the first-frame reference, leading to instruction non-compliance and hallucinated (or duplicated) visual elements. Our FG approach strengthens text–reference alignment, enabling more accurate instruction following and improved controllability. (All visual examples in this paper are from public benchmark datasets.)