DeMoGen: Towards Decompositional Human Motion Generation with Energy-Based Diffusion Models

TL;DR

DeMoGen tackles decompositional human motion generation by casting a holistic motion as a composition of $K$ motion concepts and learning their energies within an energy-based diffusion framework. It introduces three supervision regimes—DeMoGen-Exp, DeMoGen-OSS, and DeMoGen-SC—and the DeCompML dataset to enable explicit, orthogonal, and semantically-consistent concept decomposition, respectively. The approach yields superior text-to-motion and motion composition performance and supports flexible recombination of concepts to synthesize novel motions, while also providing a decomposed-motion resource that can augment training data. Overall, DeMoGen advances controllable, interpretable, and generalizable motion generation by integrating energy-based modeling with decompositional supervision and large-language-model–augmented datasets.

Abstract

Human motions are compositional: complex behaviors can be described as combinations of simpler primitives. However, existing approaches primarily focus on forward modeling, e.g., learning holistic mappings from text to motion or composing a complex motion from a set of motion concepts. In this paper, we consider the inverse perspective: decomposing a holistic motion into semantically meaningful sub-components. We propose DeMoGen, a compositional training paradigm for decompositional learning that employs an energy-based diffusion model. This energy formulation directly captures the composed distribution of multiple motion concepts, enabling the model to discover them without relying on ground-truth motions for individual concepts. Within this paradigm, we introduce three training variants to encourage a decompositional understanding of motion: 1. DeMoGen-Exp explicitly trains on decomposed text prompts; 2. DeMoGen-OSS performs orthogonal self-supervised decomposition; 3. DeMoGen-SC enforces semantic consistency between original and decomposed text embeddings. These variants enable our approach to disentangle reusable motion primitives from complex motion sequences. We also demonstrate that the decomposed motion concepts can be flexibly recombined to generate diverse and novel motions, generalizing beyond the training distribution. Additionally, we construct a text-decomposed dataset to support compositional training, serving as an extended resource to facilitate text-to-motion generation and motion composition.

Figures (8)

• Figure 1: Decompositional motion generation. (a) Our approach is able to decompose a holistic motion into multiple motion concepts with DeMoGen-Exp. This process is explicitly guided by decompositional textual descriptions, yielding diverse decomposition outcomes when varying the text cues. (b) Our approach also supports unguided decomposition using DeMoGen-OSS and DeMoGen-SC, where the model infers motion concepts without explicit decompositional text. We manually caption the inferred concepts in italic for easy understanding. (c) The decomposed motion primitives can be recombined to synthesize diverse and novel motions.
• Figure 2: Overview of our approach. We propose DeMoGen, a compositional training paradigm that facilitates decompositional motion generation via an energy-based diffusion model. We learn to decompose the holistic motion into $K$ concepts. The energy functions of these concepts are aggregated to form the $\epsilon_{pred}$, which is subsequently trained to guide the denoising process. Energy aggregation can be achieved via a denoising network and a cross-attention, supporting either latent-aware or semantic-aware modeling \ref{['sec:demogen']}. Furthermore, we investigate three variants (Section \ref{['sec:variants']}), which specify distinct strategies for learning or utilizing $\{{\bm{c}}_k\}_{k=1}^K$ within our approach.
• Figure 3: Text-to-motion generation with decompositional understanding. Given a complete textual description (in italics above the result), our approach first infers the motion concepts and further composes them to synthesize the holistic motion that matches the text. Notably, DeMoGen-OSS and DeMoGen-SC discover motion concepts without the aid of decomposed text. However, for clarity, we manually annotate each concept in bold. More visual results can be found on the https://jiro-zhang.github.io/DeMoGen/.
• Figure 4: Motion decomposition and recombination. We demonstrate that our method can infer diverse motion concepts from a complex motion sequence, conditioned on different decompositional text prompts. Our approach also exhibits the ability to recombine the inferred concepts with others to generate novel motions.
• Figure 5: Motion Diversity. From left to right, we visualize the motions generated by DeMoGen-OSS under a latent-aware setting for inferred concept 1, concept 2, and their combination.