FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space

TL;DR

FSVideo tackles the challenge of fast video diffusion by operating in a highly compressed latent space and introducing a diffusion transformer with a layer-memory mechanism to improve information flow. The framework comprises a video autoencoder (FSAE) with $64\times64\times4$ spatiotemporal compression and 128 latent channels, a diffusion transformer with Layer Memory Self-Attention and an Inter-Layer Dynamic Router, and a latent upsampler plus high-resolution refiner for multi-resolution generation. It achieves competitive video quality while delivering substantial speedups (for example, up to $42.3\times$ faster in dual-GPU setups compared to Wan2.1-I2V-14B) and demonstrates strong reconstruction and temporal coherence through Video VF loss and refined decoding strategies. The work suggests practical implications for scalable, high-quality image-to-video generation and outlines avenues for further improvement in encoding strategies, DIT design, and multimodal extensions.

Abstract

We introduce FSVideo, a fast speed transformer-based image-to-video (I2V) diffusion framework. We build our framework on the following key components: 1.) a new video autoencoder with highly-compressed latent space ($64\times64\times4$ spatial-temporal downsampling ratio), achieving competitive reconstruction quality; 2.) a diffusion transformer (DIT) architecture with a new layer memory design to enhance inter-layer information flow and context reuse within DIT, and 3.) a multi-resolution generation strategy via a few-step DIT upsampler to increase video fidelity. Our final model, which contains a 14B DIT base model and a 14B DIT upsampler, achieves competitive performance against other popular open-source models, while being an order of magnitude faster. We discuss our model design as well as training strategies in this report.

Figures (10)

• Figure 1: Videos generated via FSVideo's Image-to-Video framework while being $\mathbf{42.3\times}$ faster than Wan2.1-I2V-14B-720P. For every row, first frame is the input image from VBench huang2024vbench testset, and the following frames are generated. Zoom in to see details.
• Figure 2: Overall framework of FSVideo image-to-video pipeline. The input image is sent to the encoder to get its VAE latent code, which is then used as the condition of the base module DIT for the first diffusion process. Then, the diffused latent is sent to the upsample module, where it first passes a latent convolution neural net (CNN) upscaler, and is then combined with the input image's latent as the condition to the upsampler DIT for another diffusion process. Finally, the upsampled latent is sent to the decoder to generate output video. The decoder contains attention layers conditioned on input image's encoder feature map to enhance video quality (see Section \ref{['subsubsec:decoder_improvement']}).
• Figure 3: Overall framework of FSAE.
• Figure 4: Video reconstruction comparison between LTX-Video's autoencoder and FSAE. The first three rows represent the reconstruction results of different AE, and the last row is the ground truth. The blue box tracks the clothing textures' temporal consistency, where LTX-Video exhibits inconsistent inter-frame flickering, and FSAE consistently maintains the dotted texture. The red box compares the reconstruction quality to video details: FSAE-Lite and LTX-Video achieve comparable reconstruction results, whereas FSAE-Standard outperforms LTX-Video.
• Figure 5: Transformer layer of FSVideo.