Temporal Regularization Makes Your Video Generator Stronger

TL;DR

This work tackles the challenge of temporal quality in video generation by introducing FluxFlow, a data-level temporal augmentation that perturbs training sequences without architectural changes. FluxFlow operates in two modes—frame-level and block-level—formally defined as $V_{ ext{frame}}= ext{Shuffle}(\{F_i\igm| i\, ext{in}\,\,\ ext{S}\})+\\{F_j\bigm| j\notin\mathcal{S}\}$ with $|\,\mathcal{S}\,|=loor{\alpha N}$ and $V_{ ext{block}}=\\{B_1,...,B_M\}\$, where a subset of blocks is reordered with probability $\beta$. Implemented as training-time pre-processing, FluxFlow yields significant gains in temporal coherence and diversity across UCF-101, VBench, and multiple architectures (U-Net, DiT, AR-based), while preserving spatial fidelity. Through quantitative metrics (FVD, Subject, Flicker, Motion, Dynamic) and qualitative user studies, the authors show that frame-level perturbations generally outperform block-level ones and that FluxFlow enhances learning of temporal dynamics without compromising frame quality. The work demonstrates that simple temporal augmentation can substantially elevate video generation quality and offers a foundation for broader exploration of temporal data augmentation strategies in generative video models.

Abstract

Temporal quality is a critical aspect of video generation, as it ensures consistent motion and realistic dynamics across frames. However, achieving high temporal coherence and diversity remains challenging. In this work, we explore temporal augmentation in video generation for the first time, and introduce FluxFlow for initial investigation, a strategy designed to enhance temporal quality. Operating at the data level, FluxFlow applies controlled temporal perturbations without requiring architectural modifications. Extensive experiments on UCF-101 and VBench benchmarks demonstrate that FluxFlow significantly improves temporal coherence and diversity across various video generation models, including U-Net, DiT, and AR-based architectures, while preserving spatial fidelity. These findings highlight the potential of temporal augmentation as a simple yet effective approach to advancing video generation quality.

Figures (14)

• Figure 1: FluxFlow improves the temporal quality of video generators. Captions: (Top) A dog chasing a butterfly in a garden, with the butterfly flying in random directions. (Bottom) A person is running along a beach with waves crashing in the background.
• Figure 2: Comparison of VideoCrafter2 with FluxFlow using VBench metrics for Temporal Quality (Top) and Frame-wise and Overall Quality (Bottom). FluxFlow significantly enhances the temporal quality of generated videos while maintaining or even improving frame-wise and overall quality.
• Figure 3: Overview of FluxFlow. (a) Standard video generation trains on fixed frame orders, which may limit the model's ability to learn temporal dynamics. (b) FluxFlow introduces controlled temporal perturbations during training as a plug-and-play augmentation strategy. (c) This study explores FluxFlow at two levels: frame-level (top) and block-level (bottom). In frame-level, $\text{Num} \times 1$ denotes the number of individual frames shuffled. In block-level, $\text{Num1} \times \text{Num2}$ represents a block comprising $\text{Num2}$ consecutive frames.
• Figure 4: Illustration of FluxFlow in enhancing temporal coherence. (Top) Example frames from CogVideoX, without and with FluxFlow, showcasing larger motion dynamics in the latter. (Bottom) Comparison of temporal angle differences across frames. FluxFlow achieves consistently lower angle differences, indicating improved temporal coherence over the base model. Caption: A skateboarder performing tricks in a skatepark, with fast-paced movements and dynamic camera angles.
• Figure 5: Illustration of FluxFlow in improving temporal feature diversity. (a) Without FluxFlow, the model trained on fixed original frame sequences fails to distinguish features across different temporal paradigms. (b) With FluxFlow, features are more distinctly separated, reflecting enhanced temporal representation.