Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs

TL;DR

This paper addresses the challenge of generating directable, physically realistic humanoid motions from sparse, multi-modal inputs. It introduces the Masked Humanoid Controller (MHC), a reinforcement-learned policy paired with discriminators that handles masked directives and enables Catch-up, Combine, and Complete (CCC) behaviors. By augmenting motion data to form $\,\mathcal{M}^{+}$ and using channel- and joint-level masks, MHC learns to imitate, blend, and complete motions while remaining physically plausible under a PD-controlled physics simulation. The authors further integrate MHC with data-driven planning (FSMs and DAC-MDPs) to achieve zero-shot high-level tasks, demonstrated on a dataset of $87$ motions and across multiple modalities (MoCap, video, VR/joystick, and text). The results show improved imitation accuracy and robust planning-enabled behavior, highlighting MHC’s potential for flexible, real-time humanoid control in interactive and planning-driven settings.

Abstract

This work focuses on generating realistic, physically-based human behaviors from multi-modal inputs, which may only partially specify the desired motion. For example, the input may come from a VR controller providing arm motion and body velocity, partial key-point animation, computer vision applied to videos, or even higher-level motion goals. This requires a versatile low-level humanoid controller that can handle such sparse, under-specified guidance, seamlessly switch between skills, and recover from failures. Current approaches for learning humanoid controllers from demonstration data capture some of these characteristics, but none achieve them all. To this end, we introduce the Masked Humanoid Controller (MHC), a novel approach that applies multi-objective imitation learning on augmented and selectively masked motion demonstrations. The training methodology results in an MHC that exhibits the key capabilities of catch-up to out-of-sync input commands, combining elements from multiple motion sequences, and completing unspecified parts of motions from sparse multimodal input. We demonstrate these key capabilities for an MHC learned over a dataset of 87 diverse skills and showcase different multi-modal use cases, including integration with planning frameworks to highlight MHC's ability to solve new user-defined tasks without any finetuning.

Figures (8)

• Figure 1: Showcases generated human motions from multi-modal inputs: (A) VR device, (B) joystick controller, (C) video, and (D) text. Our proposed method, Masked Humanoid Controller (MHC), can generate physically realistic motions from a wide variety of muli-modal directives.
• Figure 2: Shows generated motions that illustrate the CCC capabilities. From left to right: MHC is able to generate motions that (1) adjust and catchup starting from an out-of-sync pose, (2) imitate a target directive that combines upper and lower body sub-segments from different motions, and (3) complete the motion from under-specified directives as indicated by missing target outlines.
• Figure 3: Illustrates the architecture and training details of the MHC framework, which consists of a controller and an ensemble of discriminators. Here the controller is trained to follow an augmented set of masked directives derived from the provided MoCap dataset. The controller gets feedback via tracking objective and style rewards generated by the ensemble of discriminators. Together they enable a directable policy to generate physically realistic motions capable of catching up, combining primitives, and completing motions from under-specified directives. (Figure layout adapted from Xu2023CompositeML)
• Figure 4: Highlights the potential applications of MHC. [Top] The selective masking of the target directive allows MHC to represent various modalities of motion data under a single framework. These multi-modal inputs include MoCap, full or occluded video, joystick, VR controller among others. [Bottom] Similarly selective masking of target directive also allows us to treat the guiding signal itself as abstract actions. This enables straightforward integration with Finite State Machines and Data Driven Planning to allow zero-shot motion generation for higher-level task specifications.
• Figure 5: Illustrates generated motions corresponding to key CCC capabilities of MHC. The simulation (left) displays key-frames of humanoid following different motion directives (right). From top to bottom the simulated humanoid (A) follows an imitation target, (B) transitions from falling-down position to catch-up to the target directive, (C) imitates motion directive that combines upper-body and lower-body movements from distinct motions (D) completes the motion using only 3D joint positions of the head, hands and feets.