Programmable Motion Generation for Open-Set Motion Control Tasks

TL;DR

The paper tackles open-set motion control by reframing arbitrary motion tasks as combinations of atomic, differentiable constraints that form an error function $F$. A pre-trained motion generator $G_ heta$ is guided by latent optimization over $z$ to minimize $F(G_ heta(z, ext{C}), p)$, enabling high-quality motions that satisfy complex, user-defined constraints. It introduces an atomic constraint library and a motion programming framework with logical operators, and notes the potential for automatic constraint generation via LLMs. Experiments demonstrate broad applicability across high-order dynamics, geometric constraints, human-scene/object interactions, self-contact, and physics-based generation, with favorable comparisons to IK and diffusion-based baselines and evidence of emergent capabilities. The approach offers a flexible, data-efficient path toward generalizable motion control applicable to AI agents and content creation, while acknowledging current limitations in constraint coverage and optimization efficiency.

Abstract

Character animation in real-world scenarios necessitates a variety of constraints, such as trajectories, key-frames, interactions, etc. Existing methodologies typically treat single or a finite set of these constraint(s) as separate control tasks. They are often specialized, and the tasks they address are rarely extendable or customizable. We categorize these as solutions to the close-set motion control problem. In response to the complexity of practical motion control, we propose and attempt to solve the open-set motion control problem. This problem is characterized by an open and fully customizable set of motion control tasks. To address this, we introduce a new paradigm, programmable motion generation. In this paradigm, any given motion control task is broken down into a combination of atomic constraints. These constraints are then programmed into an error function that quantifies the degree to which a motion sequence adheres to them. We utilize a pre-trained motion generation model and optimize its latent code to minimize the error function of the generated motion. Consequently, the generated motion not only inherits the prior of the generative model but also satisfies the required constraints. Experiments show that we can generate high-quality motions when addressing a wide range of unseen tasks. These tasks encompass motion control by motion dynamics, geometric constraints, physical laws, interactions with scenes, objects or the character own body parts, etc. All of these are achieved in a unified approach, without the need for ad-hoc paired training data collection or specialized network designs. During the programming of novel tasks, we observed the emergence of new skills beyond those of the prior model. With the assistance of large language models, we also achieved automatic programming. We hope that this work will pave the way for the motion control of general AI agents.

Figures (6)

• Figure 1: We introduce Programmable Motion Generation as a solution for open-set human motion control. Unlike previous works that treat a finite set of motion constraints as individual tasks, we attempt to solve vast and novel tasks in a unified framework. Through Programmable Motion Generation, an arbitrary controlled motion generation task is effectively solved by simply programming an error function rather than collecting training data and designing networks. The programming is also able to be implemented automatically.
• Figure 2: Overview of Programmable Motion Generation. Given an arbitrary task, we formulate it as a combination of motion constraints. Under our programming framework, by combining modules from our atomic constraint library, it is easy to program the error function to solve complex tasks just like building blocks. The programming also supports to be performed automatically by LLMs via simply providing textual descriptions of the task. Finally, the latent code $z$ of a pre-trained motion generation network is optimized to minimize the error function, thus producing motions in high quality as well as satisfying the constraints. The prompt is optional if we use text-to-motion network as the pre-trained generative model.
• Figure 3: The programming framework that pre-defines the input, output, atomic constraints and the redesigned logical operations as building blocks for motion programming. The example code corresponds to the task of "holding a ball".
• Figure 4: Qualitative examples of our method for diverse open-set motion control tasks. The task, error function code and generated motion are demonstrated for each example. The code labeled with GPT marker is generated by GPT given the task description in text.
• Figure 5: Effect of our motion prior. Top row: Ours generates valid poses while IK and IK+Reg produce invalid ones. Bottom row: IK generates incoherent motion and IK+Reg generates over-smooth motion with massive foot skating. Our method generates coherent motion while adhering to the given constraint.