GVDIFF: Grounded Text-to-Video Generation with Diffusion Models

TL;DR

GVdiff introduces grounded text-to-video generation that unifies discrete and continuous grounding using an uncertainty-based grounding injection, a spatial-temporal grounding layer, and a dynamic gate network. It leverages pre-trained T2I models to inject grounding priors and enforces temporal consistency through STGL and temporal attention, enabling robust grounding across frames. The framework supports long-range video generation, sequential prompts, and object-specific editing, outperforming baselines in temporal/prompt consistency and grounding fidelity. Limitations include reliance on language-grounding alignment and challenges with complex object interactions, pointing to future work in integrating stronger language models and more robust grounding detectors$.$

Abstract

In text-to-video (T2V) generation, significant attention has been directed toward its development, yet unifying discrete and continuous grounding conditions in T2V generation remains under-explored. This paper proposes a Grounded text-to-Video generation framework, termed GVDIFF. First, we inject the grounding condition into the self-attention through an uncertainty-based representation to explicitly guide the focus of the network. Second, we introduce a spatial-temporal grounding layer that connects the grounding condition with target objects and enables the model with the grounded generation capacity in the spatial-temporal domain. Third, our dynamic gate network adaptively skips the redundant grounding process to selectively extract grounding information and semantics while improving efficiency. We extensively evaluate the grounded generation capacity of GVDIFF and demonstrate its versatility in applications, including long-range video generation, sequential prompts, and object-specific editing.

Figures (14)

• Figure 1: Grounded text-to-Video (GVdiff) generation aims to integrate text-to-video generation with grounded generation capacity under both discrete and continuous grounding conditions, including layout, keypoint, depth map, normal map, HED map, and canny map, etc.
• Figure 2: Overview of GVdiff. (a) Connecting grounding conditions with target objects into grounded features, which are then smoothed by temporal attention. (b) Generation with Grounded-UNet, where the transformer layer of UNet is replaced with the following spatial-temporal grounding layer. (c) Spatial-temporal grounding layer. First, the uncertain-based grounding is injected into the self-attention. Then, spatial-temporal grounding attention (STGA) facilitates the interaction between the grounded features and visual tokens. An additional temporal attention layer ensures temporal consistency. Dynamic Gate Network (DGN) adaptively skips the redundant STGA.
• Figure 3: Illustration of Dynamic Gate Network (DGN), which adaptively skips redundant spatial-temporal grounding attention.
• Figure 4: Comparison with state-of-the-art-methods, including Control-A-Videl (CAV) chen2023control, ControlVideo (CV) zhang2023controlvideo, and Gen-1 esser2023structure.
• Figure 5: Skipping percentage of each layer with Dynamic Gate Network (DGN).