DIMO: Diverse 3D Motion Generation for Arbitrary Objects

TL;DR

DIMO tackles the problem of generating diverse 3D motions for arbitrary objects from a single image by distilling rich motion priors from large video models and embedding them into a shared motion latent space governed by neural key point trajectories. Motions are expressed as compact neural curves over key points and connected by a motion graph, while object geometry is captured by canonical 3D Gaussians deformed via linear blend skinning and rendered with differentiable splatting. A latent-conditioned motion decoder and KL regularization enable the joint learning of a distribution over multiple motions, allowing instant sampling, interpolation, and language-guided generation in a single forward pass. The approach achieves state-of-the-art results on 4D generation benchmarks, demonstrates strong cross-object generalization, and enables practical applications such as text-to-motion and motion reconstruction, marking a step toward a general SMPL-like paradigm for diverse dynamic objects.

Abstract

We present DIMO, a generative approach capable of generating diverse 3D motions for arbitrary objects from a single image. The core idea of our work is to leverage the rich priors in well-trained video models to extract the common motion patterns and then embed them into a shared low-dimensional latent space. Specifically, we first generate multiple videos of the same object with diverse motions. We then embed each motion into a latent vector and train a shared motion decoder to learn the distribution of motions represented by a structured and compact motion representation, i.e., neural key point trajectories. The canonical 3D Gaussians are then driven by these key points and fused to model the geometry and appearance. During inference time with learned latent space, we can instantly sample diverse 3D motions in a single-forward pass and support several interesting applications including 3D motion interpolation and language-guided motion generation. Our project page is available at https://linzhanm.github.io/dimo.

Figures (8)

• Figure 1: During inference, DIMO can instantly generate diverse 3D motions and high-fidelity 4D contents in a single forward pass from a single generative model, by sampling from a continuous motion latent space.
• Figure 2: Pipeline Overview. Given a single-view image of any general object, DIMO first distills rich motion priors from video models (Sec. \ref{['subsec:data']}). We then represent each motion as structured neural key point trajectories (Sec. \ref{['subsubsec:motion']}). During training, we embed each motion sequence into a latent code in motion latent space and jointly model diverse motion patterns using a shared motion decoder (Sec. \ref{['subsubsec:latent']}). The decoded key point transformations are used to drive canonical 3DGS for 4D optimization with only photometric losses (Sec. \ref{['subsubsec:3dgs']}).
• Figure 3: Qualitative Results. During inference, DIMO can instantly generate diverse 3D motions and photorealistic 4D contents in a single forward pass by sampling from latent space. We render three motions for each case under two views at two novel timestamps.
• Figure 4: Visual Comparison on 3D Motion Generation. DIMO can generate diverse and high-fidelity 3D motions, whereas the baseline fails to produce noticeable motions (DG4D-generated kangaroo is a thin slice so the back side appears as a mirror image of the front).
• Figure 5: Visual Comparison on Image-to-4D. DIMO can generate high-quality 4D contents for both synthetic and in-the-wild objects.