FrankenMotion: Part-level Human Motion Generation and Composition

TL;DR

FrankenMotion tackles the lack of fine-grained part-level temporal control in text-to-motion generation by constructing FrankenStein, a large-scale dataset with atomic, temporally aligned body-part annotations produced via an LLM agent. It then proposes FrankenMotion, a diffusion-based model that conditions motion generation on hierarchical prompts spanning sequence-level, atomic action, and body-part information to enable compositional and unseen motions. Experiments show that FrankenMotion outperforms adapted baselines in semantic correctness and realism, and supports flexible input control for editing and composition. The work advances fine-grained motion understanding and controllable generation, with publicly released code and data to foster broader adoption.

Abstract

Human motion generation from text prompts has made remarkable progress in recent years. However, existing methods primarily rely on either sequence-level or action-level descriptions due to the absence of fine-grained, part-level motion annotations. This limits their controllability over individual body parts. In this work, we construct a high-quality motion dataset with atomic, temporally-aware part-level text annotations, leveraging the reasoning capabilities of large language models (LLMs). Unlike prior datasets that either provide synchronized part captions with fixed time segments or rely solely on global sequence labels, our dataset captures asynchronous and semantically distinct part movements at fine temporal resolution. Based on this dataset, we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion, where each body part is guided by its own temporally-structured textual prompt. This is, to our knowledge, the first work to provide atomic, temporally-aware part-level motion annotations and have a model that allows motion generation with both spatial (body part) and temporal (atomic action) control. Experiments demonstrate that FrankenMotion outperforms all previous baseline models adapted and retrained for our setting, and our model can compose motions unseen during training. Our code and dataset will be publicly available upon publication.

Figures (4)

• Figure 1: Overview of our FrankenMotion framework. Left: Body-Part Control, where users specify fine-grained movements of individual body parts; Middle: Body-Part + Action Control, enabling coordinated whole-body actions with part-specific constraints; Right: Body-Part + Action + Sequence Control, supporting complex multi-stage motion sequences involving interactions and transitions. In all cases, FrankenAgent translates natural-language instructions into structured control signals for precise motion generation.
• Figure 2: LLM-assisted fine-grained motion annotation pipeline, compared with existing dataset. Given motions with high level action descriptions, we instruct LLM to decompose the actions into part level descriptions and align with corresponding temporal windows. This gives the most important body part text and corresponding motions needed to learn essential motion elements.
• Figure 3: Overview of our FrankenMotion. Our model is a transformer-based diffusion model that can be input conditioned on a) sequence level prompt, b) action-level prompt and c) part-level prompt. After training with our paired data of motion and structured multi-granularity text annotations, it learns the essential motion elements and how to compose them into complex motions.
• Figure 4: Qualitative comparison with baselines. Prior methods cannot compose parts into realistic motion (STMC petrovich2024stmc), generates repetitive motions (DART Zhao:DartControl:2025), or do not follow the intricate details like "turn around" (UniMotion li2024unimotion). Our method faithfully composes the complex parts into one realistic motion while also following precisely the detailed body part prompts and high level semantics.