Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop

TL;DR

This work tackles artifacts in text-to-video generation by grounding output in preconstructed 3D scenes. It introduces Scene Copilot, a training-free pipeline that combines Infinigen with LLM-driven code generation (Scene Codex) and Blender-driven interactive editing (BlenderGPT), augmented by a 323-asset procedural dataset. Empirical results show competitive video quality and superior motion consistency, with ablations confirming the value of RAG and few-shot prompts, and the ability to render long videos via human-in-the-loop control. The approach promises scalable, controllable 3D scene and video generation without heavy training, enabling rapid, higher-fidelity content creation for diverse applications.

Abstract

Video generation has achieved impressive quality, but it still suffers from artifacts such as temporal inconsistency and violation of physical laws. Leveraging 3D scenes can fundamentally resolve these issues by providing precise control over scene entities. To facilitate the easy generation of diverse photorealistic scenes, we propose Scene Copilot, a framework combining large language models (LLMs) with a procedural 3D scene generator. Specifically, Scene Copilot consists of Scene Codex, BlenderGPT, and Human in the loop. Scene Codex is designed to translate textual user input into commands understandable by the 3D scene generator. BlenderGPT provides users with an intuitive and direct way to precisely control the generated 3D scene and the final output video. Furthermore, users can utilize Blender UI to receive instant visual feedback. Additionally, we have curated a procedural dataset of objects in code format to further enhance our system's capabilities. Each component works seamlessly together to support users in generating desired 3D scenes. Extensive experiments demonstrate the capability of our framework in customizing 3D scenes and video generation.

Figures (16)

• Figure 1: Dataset Samples: A subset of assets from our dataset, accompanied by corresponding descriptions. All assets are fully procedural, meaning their components can be modified either manually or automatically.
• Figure 2: Materials in our dataset can serve as textures, and they can be automatically integrated with various assets, allowing for seamless combination and enhancing visual coherence.
• Figure 3: Various types of materials are employed as textures for a table asset, facilitating the effective fulfillment of user prompts. These materials enable seamless integration and enhance the versatility of the assets to meet diverse user requirements.
• Figure 4: Materials are combined with assets automatically, with selection guided either by user prompts or by identifying materials relevant to each specific asset. Additionally, users have the option to manually apply materials to assets according to their individual requirements.
• Figure 5: Combining procedural assets and generative models to create diverse, modular objects.