Purposer: Putting Human Motion Generation in Context

TL;DR

Purposer addresses the challenge of generating realistic human motion within 3D indoor scenes by learning a discrete latent representation of motion and training an autoregressive model in that latent space. It enables flexible conditioning through scene geometry, past observations, future targets, and semantic goals via a novel two-branch future-conditioning mechanism, allowing long-horizon, context-aware generation from short training sequences. The approach leverages unconditional data to build a strong motion prior (via PoseGPT and BABEL) and combines it with scene and semantic context to produce plausible, interactive motions, including walking paths and object interactions. Empirical results on HUMANISE and PROX show competitive performance in quality, diversity, and physical plausibility, with practical potential for indoor scene simulation and synthetic data generation.

Abstract

We present a novel method to generate human motion to populate 3D indoor scenes. It can be controlled with various combinations of conditioning signals such as a path in a scene, target poses, past motions, and scenes represented as 3D point clouds. State-of-the-art methods are either models specialized to one single setting, require vast amounts of high-quality and diverse training data, or are unconditional models that do not integrate scene or other contextual information. As a consequence, they have limited applicability and rely on costly training data. To address these limitations, we propose a new method ,dubbed Purposer, based on neural discrete representation learning. Our model is capable of exploiting, in a flexible manner, different types of information already present in open access large-scale datasets such as AMASS. First, we encode unconditional human motion into a discrete latent space. Second, an autoregressive generative model, conditioned with key contextual information, either with prompting or additive tokens, and trained for next-step prediction in this space, synthesizes sequences of latent indices. We further design a novel conditioning block to handle future conditioning information in such a causal model by using a network with two branches to compute separate stacks of features. In this manner, Purposer can generate realistic motion sequences in diverse test scenes. Through exhaustive evaluation, we demonstrate that our multi-contextual solution outperforms existing specialized approaches for specific contextual information, both in terms of quality and diversity. Our model is trained with short sequences, but a byproduct of being able to use various conditioning signals is that at test time different combinations can be used to chain short sequences together and generate long motions within a context scene.

Figures (5)

• Figure 1: An example of human motion generation in context. We propose a method able to generate realistic-looking motions that interact with virtual scenes. In this example we take a scene from ScanNet dai2017scannet. The motion can be controlled with semantic action/object queries: here the human is first commanded ' sit on table', then 'sit on couch', and finally 'lie on couch'. $\mathrm{Purposer}$ is a learning-based probabilistic model that can work efficiently with diverse types of conditioning.
• Figure 2: Method Overview. An auto-encoder is learned to compress human motion, without any context, into a discrete latent sequence space (top). A probabilistic model (bottom) is trained directly in that space, with three types of optional context: (a) scene geometry, (b) semantic goals, (c) observation of past motion.
• Figure 3: Ways of conditioning an auto-regressive model.(a): an auto-regressive model without conditioning is based on causal attention. (b): by adding a prompt token $\bm{c}_0$ to the sequence, sequence-wide conditioning can be added. (c): for time-dependent conditioning $\bm{c}_1,\hdots,\bm{c}_{T'}$, features could be combined but the model will be unaware of the future conditioning when predicting a given timestep. (d): we include future conditioning by making a non-causal network to process the time-varying conditioning, and combine their future with the standard causal generative model.
• Figure 4: Different conditionings used in $\mathrm{Purposer}$: high-level view of how different motion contexts are implemented.
• Figure 5: Effect of target pose and path conditioning.Upper row: examples of object interaction. Here we use the same object with different and random initial body position and orientation. Lower row: demonstration of the effects of path conditioning: we can define the final position and trajectory given a common starting point. The green dots represent the conditioning path.