SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion

TL;DR

SceneAdapt addresses the lack of scene context in text-conditioned motion diffusion by bridging text–motion and scene–motion data through a two-stage adaptation. Stage 1 inserts a Context-aware Keyframing (CaKey) layer to enable faithful motion inbetweening while preserving the pretrained latent manifold, and Stage 2 adds a Scene-conditioning layer with cross-attention over voxelized scene patches to inject scene-awareness. The approach preserves text fidelity and yields physically coherent motions that respect scene geometry, demonstrated via scene-aware motion generation and inbetweening experiments, with analyses revealing how spatially localized scene features guide motion. This work provides a scalable, forward-sampling solution with practical impact for realistic, scene-consistent humanoid motion generation, and sets the stage for future exploration of patch-level scene interactions in diffusion-based animation.

Abstract

Human motion is inherently diverse and semantically rich, while also shaped by the surrounding scene. However, existing motion generation approaches address either motion semantics or scene-awareness in isolation, since constructing large-scale datasets with both rich text--motion coverage and precise scene interactions is extremely challenging. In this work, we introduce SceneAdapt, a framework that injects scene awareness into text-conditioned motion models by leveraging disjoint scene--motion and text--motion datasets through two adaptation stages: inbetweening and scene-aware inbetweening. The key idea is to use motion inbetweening, learnable without text, as a proxy task to bridge two distinct datasets and thereby inject scene-awareness to text-to-motion models. In the first stage, we introduce keyframing layers that modulate motion latents for inbetweening while preserving the latent manifold. In the second stage, we add a scene-conditioning layer that injects scene geometry by adaptively querying local context through cross-attention. Experimental results show that SceneAdapt effectively injects scene awareness into text-to-motion models, and we further analyze the mechanisms through which this awareness emerges. Code and models will be released.

Figures (6)

• Figure 1: Motivation. (a) Distribution of motion embeddings extracted with the feature extractor humanml3d and visualized via PCA. HSI datasets humanisejiang2024trumans show narrower distributions than T2M dataset humanml3d, indicating lower diversity and semantic coverage. (b) Models traind on T2M datasets capture diverse action semantics but lack scene awareness, penetrating the obstacles. (c) Models trained on HSI datasets satisfy scene constraints but fail to follow text conditions.
• Figure 2: Overview. Starting from a pretrained text-to-motion model (Stage 0), we first insert CaKey layers and train them with a motion inbetweening objective (Stage 1), which only requires motion sequences. We then add scene-conditioning layers (denoted SceneCo) and train them with a scene-aware inbetweening objective (Stage 2), using scene-motion pairs. During inference, we only use the base model and ScenCo layers for scene-aware text-to-motion generation.
• Figure 3: CaKey Layer. Text-to-motion models adapted with CaKey exceed the performace of inbetweening models trained from sctratch.
• Figure 4: Qaulitative results on our evaluation set, where red boxes mark collisions and red X’s mark semantic errors; unlike AffordMotion (scene penetration) and DNO (weak text alignment), our method improves MDM by enhancing scene-awareness while preserving text fidelity.
• Figure 5: Ablation Studies on Scene-aware Inbetweening. Each dot indicates a different scene CFG weight ($w_s$) ranging from 0.001 to 2.5. Text CFG weights are fixed to 2.5.