ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way

TL;DR

ByTheWay presents a training-free booster for text-to-video diffusion models that mitigates structural and temporal artifacts while boosting motion. It comprises Temporal Self-Guidance, which aligns temporal attention across decoder blocks, and Fourier-based Motion Enhancement, which amplifies high-frequency content in temporal attention maps to increase motion magnitude; both are designed to be plug-and-play with existing backbones like AnimateDiff and VideoCrafter2. The approach is validated through qualitative and quantitative assessments, including human studies, multimodal LLM evaluations, and VBench, showing meaningful gains with negligible inference overhead. It extends to image-to-video tasks and offers a principled, energy-based framework for motion enhancement, broadening practical capabilities for high-quality video synthesis without retraining.

Abstract

The text-to-video (T2V) generation models, offering convenient visual creation, have recently garnered increasing attention. Despite their substantial potential, the generated videos may present artifacts, including structural implausibility, temporal inconsistency, and a lack of motion, often resulting in near-static video. In this work, we have identified a correlation between the disparity of temporal attention maps across different blocks and the occurrence of temporal inconsistencies. Additionally, we have observed that the energy contained within the temporal attention maps is directly related to the magnitude of motion amplitude in the generated videos. Based on these observations, we present ByTheWay, a training-free method to improve the quality of text-to-video generation without introducing additional parameters, augmenting memory or sampling time. Specifically, ByTheWay is composed of two principal components: 1) Temporal Self-Guidance improves the structural plausibility and temporal consistency of generated videos by reducing the disparity between the temporal attention maps across various decoder blocks. 2) Fourier-based Motion Enhancement enhances the magnitude and richness of motion by amplifying the energy of the map. Extensive experiments demonstrate that ByTheWay significantly improves the quality of text-to-video generation with negligible additional cost.

Figures (20)

• Figure 1: Unlock the potential of pretrained text-to-video (T2V) generation models in a training-free approach. (1) ByTheWay helps to enhance structural plausibility and temporal consistency in generated videos, significantly reducing artifacts and flickering. (2) ByTheWay contributes to enriching motion patterns and amplifying the motion magnitude in generated videos. Further, ByTheWay can be seamlessly integrated into various powerful T2V backbones (e.g., AnimateDiffguo2023animatediff and VideoCrafter2chen2024videocrafter2) in a plug-and-play manner, serving as a highly extensible module without introducing additional parameters or sampling cost.
• Figure 2: Statistical patterns derived from T2V generation process. (a) Generated videos exhibiting structurally implausible and temporally inconsistent artifacts demonstrate greater disparity between the temporal attention maps of different decoder blocks. (b) After applying ByTheWay, the modeling disparity in original corrupted videos are reduced to the level of well-generated videos. (c) Videos with larger motion magnitude typically exhibit higher energy, in which the motion magnitude is measured by the estimated optical flow.
• Figure 3: Temporal Self-Guidance. Temporal Self-Guidance contributes to the restoration of collapsed structures and consistency of motion in the generated video.
• Figure 4: Energy representation of video motion magnitude. Samples with richer motion typically exhibit a higher energy.
• Figure 5: Frequency decomposition. By directly removing either the high-frequency or low-frequency components from the temporal attention map, it can be observed that motion in generated videos is primarily present in the high-frequency components.