TSTMotion: Training-free Scene-aware Text-to-motion Generation

TL;DR

This work tackles scene-aware text-to-motion generation without requiring scene-grounded training data by introducing TSTMotion, a training-free framework that leverages foundation models to bridge 3D scenes and text descriptions. It builds a four-component pipeline—Scene Compiler, Motion Planner, Aligned Motion Diffusion Models, and Motion Checker—to craft and refine a scene-consistent motion guidance that can steer blank-background Motion Diffusion Models. Two training-free modifications align the diffusion outputs to the guidance and penalize overlaps with the 3D scene, enabling realistic, scene-interacting motions without dataset-specific training. Experiments on HUMANISE and AffordMotion demonstrate superior or competitive performance and strong generalization to unseen outdoor scenes, with ablations validating the contribution of each component. The approach offers practical impact by enabling scalable, diverse, and controllable scene-aware motion generation without collecting costly scene-motion data, and the authors release the code for public use.

Abstract

Text-to-motion generation has recently garnered significant research interest, primarily focusing on generating human motion sequences in blank backgrounds. However, human motions commonly occur within diverse 3D scenes, which has prompted exploration into scene-aware text-to-motion generation methods. Yet, existing scene-aware methods often rely on large-scale ground-truth motion sequences in diverse 3D scenes, which poses practical challenges due to the expensive cost. To mitigate this challenge, we are the first to propose a \textbf{T}raining-free \textbf{S}cene-aware \textbf{T}ext-to-\textbf{Motion} framework, dubbed as \textbf{TSTMotion}, that efficiently empowers pre-trained blank-background motion generators with the scene-aware capability. Specifically, conditioned on the given 3D scene and text description, we adopt foundation models together to reason, predict and validate a scene-aware motion guidance. Then, the motion guidance is incorporated into the blank-background motion generators with two modifications, resulting in scene-aware text-driven motion sequences. Extensive experiments demonstrate the efficacy and generalizability of our proposed framework. We release our code in \href{https://tstmotion.github.io/}{Project Page}.

Figures (3)

• Figure 1: Illustration of scene-aware text-driven motion sequences generated by our TSTMotion framework in different 3D scenes based on text descriptions without any training. For clarity, as time progresses, human avatars in motion sequences transition from light to dark colors. More qualitative results in image and video formats are available in the supplementary.
• Figure 2: An overview of our proposed training-free TSTMotion framework for the given text $d$ and 3D scene $S_{3D}$. At first, the Scene Compiler extracts the spatial auxiliary in the $S_{3D}$. Based on the spatial auxiliary, the Motion Planner incorporates the text description and well-designed prompt templates to infer the motion guidance $s[M_{mask}]$. Equipped with the $s[M_{mask}]$, the Aligned Motion Diffusion Model predicts initial scene-aware text-driven motion sequences $m$ with two training-free modifications. Finally, the Motion Checker is applied to iteratively refine and generate the final $m$ to better align with the $d$ and $S_{3D}$.
• Figure 3: Comparison of between Wang et al.wang2022humanise, DIMOS*, AffordMotion and our TSTMotion on the unseen PROX dataset.