Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions

TL;DR

The paper investigates the role of CLIP image embeddings in Stable Video Diffusion for image-to-video generation and finds that while CLIP enhances aesthetics, it does not improve subject or background consistency. It shows that Temporal Cross-Attention is unnecessary and Spatial Cross-Attention can be replaced by a one-time linear layer computed at the first step and cached for the rest of inference, enabling VCUT, a training-free efficiency method. VCUT eliminates Temporal Cross Attention and substitutes Spatial Cross-Attention with a simple linear layer, achieving up to 322T MACs and up to 50M fewer parameters, with about 20% faster latency, all without additional training. The method is driven by a two-stage inference framework—Semantic Binding and Quality Improvement—where conditioning during Semantic Binding suffices, significantly reducing compute while preserving video quality and consistency.

Abstract

This paper investigates the role of CLIP image embeddings within the Stable Video Diffusion (SVD) framework, focusing on their impact on video generation quality and computational efficiency. Our findings indicate that CLIP embeddings, while crucial for aesthetic quality, do not significantly contribute towards the subject and background consistency of video outputs. Moreover, the computationally expensive cross-attention mechanism can be effectively replaced by a simpler linear layer. This layer is computed only once at the first diffusion inference step, and its output is then cached and reused throughout the inference process, thereby enhancing efficiency while maintaining high-quality outputs. Building on these insights, we introduce the VCUT, a training-free approach optimized for efficiency within the SVD architecture. VCUT eliminates temporal cross-attention and replaces spatial cross-attention with a one-time computed linear layer, significantly reducing computational load. The implementation of VCUT leads to a reduction of up to 322T Multiple-Accumulate Operations (MACs) per video and a decrease in model parameters by up to 50M, achieving a 20% reduction in latency compared to the baseline. Our approach demonstrates that conditioning during the Semantic Binding stage is sufficient, eliminating the need for continuous computation across all inference steps and setting a new standard for efficient video generation.

Figures (3)

• Figure 1: Illustration of the limitation of the CLIP model, which inappropriately assigns high similarity scores despite significant changes in perspective between video frames, suggesting a lack of sensitivity to visual variations. Conversely, the DINO oquab2024dinov model more accurately reflects changes, showing lower similarity scores for larger variations.
• Figure 2: This figure illustrates various applications of CLIP image embedding across different stages of the diffusion process. The top left panel shows CLIP image embedding applied at all steps. The top right panel applies it only during the Quality Improvement stage (later stages), while the bottom left panel uses it exclusively during the Semantic Binding stage (early steps). The bottom right panel does not apply CLIP image embedding at any stage of the diffusion process. This comparison demonstrates that while image embeddings significantly influence the generation process in early stages, their impact lessens in later steps, suggesting that it is feasible to omit embeddings in advanced stages without a loss in image quality.
• Figure 3: The figure showcases video frames generated by standard SVD and VCUT-integrated SVD models. The top two rows demonstrate enhanced motion in the shark video, highlighting the benefits of VCUT in dynamic degree metric as noted in Table \ref{['tab:different_steps_quality']}. Middle rows show that removing TCA improves the consistency of teapot videos, indicated by red boxes. The bottom rows confirm that VCUT does not compromise spatial quality, as seen in the less blurry butterfly frames. These comparisons illustrate VCUT's effectiveness in enhancing video dynamics and quality without additional computational costs.