Choreographing a World of Dynamic Objects

TL;DR

CHORD tackles the challenge of generating 4D scene-level motion for scenes with multiple interacting objects by distilling dynamics from 2D video generative models. It introduces a Score Distillation Sampling framework extended to 4D, paired with a Rectified Flow–based distillation target and a domain-tailored noise schedule. The method employs a hierarchical 4D representation that combines bi-level spatial control points with a Fenwick-tree temporal encoding, along with temporal and spatial regularization to stabilize optimization. The authors demonstrate robust, scalable 4D motion generation and show applicability to robotic manipulation and long-horizon dynamics, outperforming several baselines on prompt adherence and realism. This work suggests a practical path toward scalable, category-agnostic 4D synthesis guided by powerful video priors, with significant implications for robotics and embodied AI.

Abstract

Dynamic objects in our physical 4D (3D + time) world are constantly evolving, deforming, and interacting with other objects, leading to diverse 4D scene dynamics. In this paper, we present a universal generative pipeline, CHORD, for CHOReographing Dynamic objects and scenes and synthesizing this type of phenomena. Traditional rule-based graphics pipelines to create these dynamics are based on category-specific heuristics, yet are labor-intensive and not scalable. Recent learning-based methods typically demand large-scale datasets, which may not cover all object categories in interest. Our approach instead inherits the universality from the video generative models by proposing a distillation-based pipeline to extract the rich Lagrangian motion information hidden in the Eulerian representations of 2D videos. Our method is universal, versatile, and category-agnostic. We demonstrate its effectiveness by conducting experiments to generate a diverse range of multi-body 4D dynamics, show its advantage compared to existing methods, and demonstrate its applicability in generating robotics manipulation policies. Project page: https://yanzhelyu.github.io/chord

Figures (14)

• Figure 1: 4D scene motion generated by our method. We present Chord, a universal generative pipeline capable of animating scenes with multiple objects that interact with each other. Project page: https://yanzhelyu.github.io/chord
• Figure 2: Overview. For the input meshes of a given scene, we first convert them into 3D-GS representations to enable smooth gradient computation. The converted 3D-GS models are then used to initialize a 4D representation (Sec. \ref{['sec:4D-representation']}). We iteratively refine this 4D representation by sampling camera poses at each iteration, rendering the corresponding videos, and passing them to the video generation model to obtain optimization gradients (Sec. \ref{['sec:supervision']}). Additionally, we compute regularization terms (Sec. \ref{['sec:regularization']}) to enforce spatial and temporal smoothness during the optimization process.
• Figure 3: Illustration of the hierarchical control point representation. We represent the deformation using a spatial hierarchical structure. Coarse control points capture large-scale deformations, while fine control points refine local details.
• Figure 4: Illustration of the Fenwick Tree representation. Each node stores the cumulative deformation over a temporal range, allowing nearby frames to share parameters and naturally enforcing temporal coherence. For example, $(r_k^{[6]}, T_k^{[6]})$ encodes the accumulated deformation from frames 5–6. Queries for frames 6 and 7 then compose their deformations from a small, overlapping set of nodes, as shown in the figure.
• Figure 5: Qualitative comparisons. We compare our approach with several mesh animation methods. Our method produces results that better align with the given prompts and exhibit more natural motion. In the figure, A3D refers to Animate3D jiang2024animate3d, AAM denotes AnimateAnyMesh wu2025animateanymesh, MD represents MotionDreamer uzolas2025motiondreamer, and TC corresponds to 4D reconstruction results from videos generated by TrajectoryCrafter Yu_2025_ICCV. For additional comparisons and full animation results, please refer to our supplementary website.