FreePCA: Integrating Consistency Information across Long-short Frames in Training-free Long Video Generation via Principal Component Analysis

TL;DR

This work tackles the challenge of generating long videos from diffusion models trained on short clips by addressing distribution shifts across extended frame counts. It introduces FreePCA, a training-free approach that uses Principal Component Analysis to decouple global appearance consistency from local motion information, enabling a principled integration of global consistency with high-quality local detail. The method comprises Consistency Feature Decomposition to extract alignment-friendly components and Progressive Fusion to progressively inject them into sliding-window local features, complemented by reusing the mean statistics of the initial noise for stability. Empirical results across multiple base models and prompts show FreePCA achieves superior consistency and quality without additional training, with demonstrated applicability to multi-prompt and continuing generation scenarios.

Abstract

Long video generation involves generating extended videos using models trained on short videos, suffering from distribution shifts due to varying frame counts. It necessitates the use of local information from the original short frames to enhance visual and motion quality, and global information from the entire long frames to ensure appearance consistency. Existing training-free methods struggle to effectively integrate the benefits of both, as appearance and motion in videos are closely coupled, leading to motion inconsistency and visual quality. In this paper, we reveal that global and local information can be precisely decoupled into consistent appearance and motion intensity information by applying Principal Component Analysis (PCA), allowing for refined complementary integration of global consistency and local quality. With this insight, we propose FreePCA, a training-free long video generation paradigm based on PCA that simultaneously achieves high consistency and quality. Concretely, we decouple consistent appearance and motion intensity features by measuring cosine similarity in the principal component space. Critically, we progressively integrate these features to preserve original quality and ensure smooth transitions, while further enhancing consistency by reusing the mean statistics of the initial noise. Experiments demonstrate that FreePCA can be applied to various video diffusion models without requiring training, leading to substantial improvements. Code is available at https://github.com/JosephTiTan/FreePCA.

Figures (9)

• Figure 1: Illustration of different training-free methods for generating long videos. (a) Global aligned method, which inputs the entire video sequence into the model, resulting in lower quality, object loss, and slow motion, but maintains consistency. (b) Local stitched method, which uses a sliding window to extract video segments along the temporal dimension, resulting in poor consistency but retaining the original generation quality. (c) Our FreePCA, which effectively combines the global and local methods via PCA, achieving good consistency while preserving the original generation quality.
• Figure 2: (a-c) Illustration of consistency information extraction after applying PCA to videos and converting each component's information into edge maps. After applying PCA to video (a), some components' information exhibit high consistency like (b), while others show inconsistency like (c). We use PSNR to assess whether each component's information exhibit consistency as the original video, setting a threshold of 35 dB. (d) shows the statistics of 100 videos generated by the global and local methods. We categorize these videos based on whether they have consistency components into two groups: high consistency, and low consistency. It indicates that PCA can separate consistency information and exhibits statistical patterns, reflecting different consistency degree of different methods in the principal component space.
• Figure 3: Visualization of consistency features extracted in the principal component space using cosine similarity. After applying both global and local methods in temporal attention and extracting features from the principal component space, we subtract each adjacent frame to exhibit the intensity of changes over time. (a) and (b) show features across all components. (c), (d), (e), and (f) illustrate features selected based on the cosine similarity between global and local features. The distinct character outlines in (c) and (d) indicate that features with high cosine similarity have consistent appearance attributes, while the significant intensity difference between (e) and (f) indicates that features with low cosine similarity exhibit motion intensity attributes.
• Figure 4: Overview of our method. For noise initialization, we extend short initial noise into long initial noise using a reuse mean statistics approach. In the iterative denoising process, we fuse global and local features in temporal attention through two processes: Consistency Feature Decomposition and Progressive Fusion. For Consistency Feature Decomposition, we crop the global features to the same size as the local features based on their positions. Then, we apply PCA along the temporal dimension to both, using cosine similarity to compare global and local features and decouple them into consistent appearance and motion intensity features. Finally, we perform Progressive Fusion to integrate the global consistent features into the local motion features progressively (see \ref{['fig:ppl2']} for details).
• Figure 5: Illustration of Progressive Fusion. As the window slides, the proportion of consistent appearance features from the global features is gradually increased, while the proportion of local motion intensity features is decreased. Finally, the overlapping portions within the window are averaged to obtain the final result.