SemanticGen: Video Generation in Semantic Space

TL;DR

SemanticGen tackles slow convergence and drift in long-video diffusion methods by first generating compact semantic representations and then mapping them to VAE latents in a two-stage diffusion pipeline. By compressing semantic space with a learnable MLP and using in-context conditioning, the approach achieves faster training and better long-video coherence, while extending naturally to longer videos via Swin-attention in the latent stage. Experiments show competitive short-video quality and clear improvements in long-video consistency, with ablations validating the benefits of semantic-space modeling. The work advances practical, scalable text-to-video generation by decoupling global semantic planning from detail synthesis, enabling efficient generation of longer videos with high fidelity.

Abstract

State-of-the-art video generative models typically learn the distribution of video latents in the VAE space and map them to pixels using a VAE decoder. While this approach can generate high-quality videos, it suffers from slow convergence and is computationally expensive when generating long videos. In this paper, we introduce SemanticGen, a novel solution to address these limitations by generating videos in the semantic space. Our main insight is that, due to the inherent redundancy in videos, the generation process should begin in a compact, high-level semantic space for global planning, followed by the addition of high-frequency details, rather than directly modeling a vast set of low-level video tokens using bi-directional attention. SemanticGen adopts a two-stage generation process. In the first stage, a diffusion model generates compact semantic video features, which define the global layout of the video. In the second stage, another diffusion model generates VAE latents conditioned on these semantic features to produce the final output. We observe that generation in the semantic space leads to faster convergence compared to the VAE latent space. Our method is also effective and computationally efficient when extended to long video generation. Extensive experiments demonstrate that SemanticGen produces high-quality videos and outperforms state-of-the-art approaches and strong baselines.

Figures (14)

• Figure 1: Examples synthesized by SemanticGen. SemanticGen generates high-quality videos from text prompts in the semantic representation space and scales to long-form generation of up to one-minute videos. Video results are on our https://jianhongbai.github.io/SemanticGen/.
• Figure 2: Illustration of the proposed SemanticGen.
• Figure 3: Overview of SemanticGen. (a) We train a semantic generator to fit the compressed semantic representation distribution of off-the-shelf semantic encoders. (b) We optimized a latent diffusion model for denoising video VAE latents conditioned on their semantic representations. (c) During inference, we integrate the semantic generator and VAE latent generator to achieve high-quality T2V generation.
• Figure 4: Video generation conditioned on semantic features extracted from a reference video.Row 1: The reference video. Rows 2–4: Reconstructions based on semantic representations (Sem. Rep.) with dimensions of 2048, 64, and 8, respectively. Row 5: T2V generation results without semantic representations.
• Figure 5: Implementation of Swin-Attention. When generating long videos, we apply full attention to model the semantic representations and use shifted-window attention swin to map them into the VAE space. The blue squares indicate VAE latents, while the yellow squares denote semantic representations.