Mobius: A High Efficient Spatial-Temporal Parallel Training Paradigm for Text-to-Video Generation Task

TL;DR

Text-to-video generation suffers from high computational costs due to serial gradient flow in 3D-Unet-based architectures. Mobius introduces a spatial-temporal parallel training paradigm that decouples and parallelizes spatial and temporal branches, freezing spatial layers while training temporal ones. On a cartoon emoji dataset, Mobius achieves 24% memory savings and 12% training time reduction with video outputs that are comparable in quality to serial baselines. This approach provides a practical and scalable direction for efficient fine-tuning of diffusion-based T2V models in AIGC scenarios.

Abstract

Inspired by the success of the text-to-image (T2I) generation task, many researchers are devoting themselves to the text-to-video (T2V) generation task. Most of the T2V frameworks usually inherit from the T2I model and add extra-temporal layers of training to generate dynamic videos, which can be viewed as a fine-tuning task. However, the traditional 3D-Unet is a serial mode and the temporal layers follow the spatial layers, which will result in high GPU memory and training time consumption according to its serial feature flow. We believe that this serial mode will bring more training costs with the large diffusion model and massive datasets, which are not environmentally friendly and not suitable for the development of the T2V. Therefore, we propose a highly efficient spatial-temporal parallel training paradigm for T2V tasks, named Mobius. In our 3D-Unet, the temporal layers and spatial layers are parallel, which optimizes the feature flow and backpropagation. The Mobius will save 24% GPU memory and 12% training time, which can greatly improve the T2V fine-tuning task and provide a novel insight for the AIGC community. We will release our codes in the future.

Figures (6)

• Figure 1: Existing tuning paradigm in T2V generation task. The top is the full fine-tuning which goes through all parameters. The bottom is Delta-Tuning which only update part parameters. represents the frozen part, and represents the trainable part.
• Figure 2: The Mobius framework introduces a spatial-temporal parallel paradigm for text-to-video tasks. Existing text-to-video works often add extra serial temporal layers to ensure continuous action, which can preserve unnecessary gradients. However, the Mobius framework devises a parallel mode for spatial and temporal layers. This means that the gradient only flows in the inner ring, which corresponds to the temporal layers. This approach aims to improve the efficiency and effectiveness of text-to-video generation tasks. During the training process, the spatial layers are frozen, and the temporal layers are trainable.
• Figure 3: The framework for text-to-video conversion consists of a text-encoder, a VAE, and a 3D-Unet. The pie chart shows that the majority of the model parameters are contained within the 3D-Unet.
• Figure 4: Existing tuning paradigm in text-generate video task. (a) is the Full fine-tuning mode. (b) and (c) are Delta-tuning. (b) is the serial mode mode, and (c) is our parallel mode. The black arrow means the forward step, and the red arrow means the backward step. Both modes (a) and (b) need to reserve and compute all the gradients. However, our parallel mode only needs to reserve and compute the temporal layers gradient, which is highly efficient. represents the frozen part, and represents the trainable part.
• Figure 5: Prompt words (Chinese are translated to English for reading convenience.) and their corresponding generated video