Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autonomous Character-Scene Interaction Synthesis from Text Instruction

Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, Yixin Zhu

TL;DR

This work addresses language-guided synthesis of multi-stage, scene-aware human motion in 3D environments by proposing a unified pipeline that generates motion segments autonomously from a single text instruction and a goal location. It combines an auto-regressive diffusion model with a dual voxel scene encoder and a frame-embedded text conditioning, guided by an autonomous scheduler to transition between locomotion, reaching, and interaction stages. A VR-assisted mocap dataset, LINGO, provides 16 hours of richly annotated motions across 120 indoor scenes and 40 action types, enabling training of the proposed framework. Experimental results across locomotion, object reaching, and interactive motion demonstrate improved scene awareness, reduced scene penetration, and coherent, semantically aligned motions compared to baselines like TRUMANS, while ablations underscore the importance of frame embedding and the dual-voxel representation. Overall, the approach advances autonomous, text-driven animation by tightly coupling language, motion diffusion, and scene geometry, with practical impact for real-time, scene-consistent character animation.

Abstract

Synthesizing human motions in 3D environments, particularly those with complex activities such as locomotion, hand-reaching, and human-object interaction, presents substantial demands for user-defined waypoints and stage transitions. These requirements pose challenges for current models, leading to a notable gap in automating the animation of characters from simple human inputs. This paper addresses this challenge by introducing a comprehensive framework for synthesizing multi-stage scene-aware interaction motions directly from a single text instruction and goal location. Our approach employs an auto-regressive diffusion model to synthesize the next motion segment, along with an autonomous scheduler predicting the transition for each action stage. To ensure that the synthesized motions are seamlessly integrated within the environment, we propose a scene representation that considers the local perception both at the start and the goal location. We further enhance the coherence of the generated motion by integrating frame embeddings with language input. Additionally, to support model training, we present a comprehensive motion-captured dataset comprising 16 hours of motion sequences in 120 indoor scenes covering 40 types of motions, each annotated with precise language descriptions. Experimental results demonstrate the efficacy of our method in generating high-quality, multi-stage motions closely aligned with environmental and textual conditions.

Autonomous Character-Scene Interaction Synthesis from Text Instruction

TL;DR

This work addresses language-guided synthesis of multi-stage, scene-aware human motion in 3D environments by proposing a unified pipeline that generates motion segments autonomously from a single text instruction and a goal location. It combines an auto-regressive diffusion model with a dual voxel scene encoder and a frame-embedded text conditioning, guided by an autonomous scheduler to transition between locomotion, reaching, and interaction stages. A VR-assisted mocap dataset, LINGO, provides 16 hours of richly annotated motions across 120 indoor scenes and 40 action types, enabling training of the proposed framework. Experimental results across locomotion, object reaching, and interactive motion demonstrate improved scene awareness, reduced scene penetration, and coherent, semantically aligned motions compared to baselines like TRUMANS, while ablations underscore the importance of frame embedding and the dual-voxel representation. Overall, the approach advances autonomous, text-driven animation by tightly coupling language, motion diffusion, and scene geometry, with practical impact for real-time, scene-consistent character animation.

Abstract

Synthesizing human motions in 3D environments, particularly those with complex activities such as locomotion, hand-reaching, and human-object interaction, presents substantial demands for user-defined waypoints and stage transitions. These requirements pose challenges for current models, leading to a notable gap in automating the animation of characters from simple human inputs. This paper addresses this challenge by introducing a comprehensive framework for synthesizing multi-stage scene-aware interaction motions directly from a single text instruction and goal location. Our approach employs an auto-regressive diffusion model to synthesize the next motion segment, along with an autonomous scheduler predicting the transition for each action stage. To ensure that the synthesized motions are seamlessly integrated within the environment, we propose a scene representation that considers the local perception both at the start and the goal location. We further enhance the coherence of the generated motion by integrating frame embeddings with language input. Additionally, to support model training, we present a comprehensive motion-captured dataset comprising 16 hours of motion sequences in 120 indoor scenes covering 40 types of motions, each annotated with precise language descriptions. Experimental results demonstrate the efficacy of our method in generating high-quality, multi-stage motions closely aligned with environmental and textual conditions.
Paper Structure (49 sections, 1 equation, 9 figures, 4 tables)

This paper contains 49 sections, 1 equation, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interaction Synthesis in Static Scenes
  4. Interaction Synthesis for Dynamic Objects
  5. Character-scene Interaction Synthesis Datasets
  6. Method
  7. Data Representation
  8. Human Motion
  9. Conditions
  10. Motion Diffusion Module
  11. Dual Voxel Scene Encoder
  12. Frame-embedded Text Encoder
  13. Stage-specific Goal
  14. Autonomous Scheduler
  15. LINGO dataset
  16. ...and 34 more sections

Figures (9)

  • Figure 1: Overview of our method. Our method uses an auto-regressive diffusion model that generates the next motion segment based on existing motions (\ref{['sec:method_diffusion']}). The 3D environment is captured through a dual voxel scene encoder (\ref{['sec:method_scene_embed']}). The text instructions are encoded with the time frame to provide precise and time-specific semantic guidance (\ref{['sec:method_text_embed']}). The goal encoder (\ref{['sec:method_goal_embed']}) embeds the sub-goal locations for different interaction stages, which are automatically determined by our autonomous scheduler (\ref{['sec:method_scheduler']}).
  • Figure 2: Comparison results. We qualitatively compare our method with TRUMANS jiang2024scaling. The left side shows the locomotion along a trajectory, and the right side shows the interaction of sitting on the sofa. Our method generates characters that actively avoid penetrating the scene and exhibit natural cues of scene awareness. For more qualitative results, we refer readers to the supplementary video.
  • Figure 3: LINGO dataset. We show some selected frames and the setup of the VR-assisted mocap.
  • Figure 4: Qualitative comparison. We compare (a) our method with (b) TRUMANS jiang2024scaling on the task of walking to the goal location. It is shown that our method is aware of the surroundings for collision avoidance, while TRUMANS depends on a pre-defined trajectory. We show (c) our method and (d) w/o frame embedder given "grasp an object" instruction. The synthesized motion without a frame embedder is disordered and tends to repeat.
  • Figure 5: Number of occurrences of each motion type in LINGO dataset.
  • ...and 4 more figures