Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compositional 3D-aware Video Generation with LLM Director

Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian

TL;DR

A novel paradigm is proposed that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models and makes the generated frames adhere to natural image distribution.

Abstract

Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \url{https://aka.ms/c3v}.

Compositional 3D-aware Video Generation with LLM Director

TL;DR

A novel paradigm is proposed that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models and makes the generated frames adhere to natural image distribution.

Abstract

Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \url{https://aka.ms/c3v}.
Paper Structure (29 sections, 8 equations, 5 figures, 1 table)

This paper contains 29 sections, 8 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Video Generation with LLM
  4. Compositional 3D Generation
  5. Preliminaries
  6. 3D Gaussian Splatting
  7. Score Distillation Sampling
  8. Method
  9. Overview.
  10. Task Decomposition with LLM
  11. Task Instructions.
  12. 3D Representation.
  13. Instantiation.
  14. Coarse-grained Trajectory Generation with LLM
  15. Fine-grained Composition with 2D Diffusion Priors
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Illustration of our method. It consists of three stages: 1) The input textual prompt is decomposed into individual concepts by the LLM. Then we generate each concept in the form of 3D with the corresponding pre-trained expert model (left & Sec. \ref{['LLM-based Task Decomposition']}). 2) We leverage knowledge in multi-modal LLM to estimate the 2D trajectory of objects step-by-step (middle & Sec. \ref{['GPT-4V-based Trajectory Estimation']}). 3) After lifting the estimated 2D trajectory into 3D as initialization, we refine the scales, locations, and rotations of objects within the 3D scene using 2D diffusion priors (right & Sec. \ref{['SDS-based Refinement']}).
  • Figure 2: Illustration of coarse-grained trajectory generation with LLM. Instead of querying multi-modal LLM to estimate dynamic trajectory directly, we generate trajectory in a step-by-step manner: estimating the locations of starting and ending points first, then reasoning the path between them.
  • Figure 3: Qualitative comparisons with baselines. When prompting complex queries, the baseline methods fail to follow the queries in terms of the number of objects and the corresponding motion. In contrast, our method excels in yielding both diverse motion and high visual quality.
  • Figure 4: Ablation studies on framework design. Each ablation is prompted with the same text.
  • Figure 5: Our method offers flexible control of individual concepts. We demonstrate this by editing different concepts: the appearance and motion of the actors, and the scenes.