TC4D: Trajectory-Conditioned Text-to-4D Generation

TL;DR

TC4D addresses the limitation of existing 4D generation methods that confine motion to local, bounded regions by decoupling motion into global and local components and using trajectory conditioning. It introduces a trajectory-driven global rigid transform of the scene's bounding box and a trajectory-aware VSDS-based local deformation learned from a pre-trained text-to-video model, enabling animation along arbitrary trajectories and compositional scenes. The method uses a deformable NeRF representation and hash-grid-based deformation fields, with an annealed VSDS training regime to achieve temporally coherent motion. Through user studies and ablations, TC4D achieves substantially more motion and realism than baselines like 4D-fy and DreamGaussian4D, and demonstrates applicability to end-to-end automated trajectory generation. The work highlights practical impact for controllable, large-scale 4D content and points to future avenues in multi-object motion and automated metric development.

Abstract

Recent techniques for text-to-4D generation synthesize dynamic 3D scenes using supervision from pre-trained text-to-video models. However, existing representations for motion, such as deformation models or time-dependent neural representations, are limited in the amount of motion they can generate-they cannot synthesize motion extending far beyond the bounding box used for volume rendering. The lack of a more flexible motion model contributes to the gap in realism between 4D generation methods and recent, near-photorealistic video generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D generation, which factors motion into global and local components. We represent the global motion of a scene's bounding box using rigid transformation along a trajectory parameterized by a spline. We learn local deformations that conform to the global trajectory using supervision from a text-to-video model. Our approach enables the synthesis of scenes animated along arbitrary trajectories, compositional scene generation, and significant improvements to the realism and amount of generated motion, which we evaluate qualitatively and through a user study. Video results can be viewed on our website: https://sherwinbahmani.github.io/tc4d.

Figures (12)

• Figure 1: Scenes generated using trajectory-conditioned 4D generation (TC4D). We show scenes consisting of multiple dynamic objects generated with text prompts and composited together. The scene is shown for different viewpoints (panels) and at different time steps (horizontal dimension). Motion is synthesized by animating the scene bounding box along a provided trajectory using a rigid transformation, and we optimize for local deformations that are consistent with the trajectory using supervision from a video diffusion model. Overall, our approach improves the amount and realism of motion in generated 4D scenes.
• Figure 2: Comparing text-to-video and text-to-4D generation. Existing methods for video generation (e.g., VideoCrafter2 chen2024videocrafter2) create realistic objects with motion at both global and local scales. For example, a generated unicorn or corgi moves across the camera field of view (global motion) in concert with the animation of its legs and body (local motion). However, existing text-to-4D methods bahmani20234d only generate a limited amount of local motion and do not learn global motion in 3D space.
• Figure 3: Overview of TC4D. Our method takes as input a pre-trained 3D scene generated using supervision from a 2D and/or 3D diffusion model. We animate the scene through a decomposition of motion at global and local scales. Motion at the global scale is incorporated via rigid transformation of the bounding box containing the object representation along a given spline trajectory $\mathcal{T}$ at steps $t$. We align local motion to the trajectory by optimizing a separate deformation model that warps the underlying volumetric representation based on supervision from a text-to-video model. The output is an animated 3D scene with motion that is more realistic and greater in magnitude than previous techniques.
• Figure 4: Comparison of TC4D and 4D-fy. We show two generated scenes of "batman riding a camel" and "a deer walking". Each panel contains images rendered from a single viewpoint from two steps along the same trajectory. While 4D-fy produces mostly static results (which we rigidly transform along the trajectory), TC4D learns coherent motion and produces a walking animation. Please also refer to the video results in the supplement.
• Figure 5: Trajectory with scale. We demonstrate adding a scale term to the trajectory for a scene generated with the text prompt "a flame getting larger". We compare generating the scene without a trajectory or scale (top), with an upwards trajectory only (middle), and with both a trajectory and scale (bottom), the last of which yields a convincing result.