Dynamic Worlds, Dynamic Humans: Generating Virtual Human-Scene Interaction Motion in Dynamic Scenes

TL;DR

Dyn-HSI introduces a world-model–inspired cognitive framework for dynamic human–scene interaction by integrating a Dynamic Scene-Aware Navigation Vision module, Hierarchical Experience Memory, and a multimodal Human–Scene Interaction Diffusion Model with a Task-adaptive Condition Adapter. It extends static HSI evaluation to the Dyn-Scenes dynamic benchmark, and demonstrates superior motion fidelity, trajectory realism, and reduced penetration artifacts in both static and dynamic environments, including out-of-distribution scenarios. The approach leverages autoregressive diffusion, memory-primed initialization, and adaptive conditioning to maintain scene-aware interactions as scenes evolve. The work provides a practical pathway toward immersive, robust virtual humans that can operate in continuously changing environments, with substantial implications for VR, simulation, and robotics planning.

Abstract

Scenes are continuously undergoing dynamic changes in the real world. However, existing human-scene interaction generation methods typically treat the scene as static, which deviates from reality. Inspired by world models, we introduce Dyn-HSI, the first cognitive architecture for dynamic human-scene interaction, which endows virtual humans with three humanoid components. (1)Vision (human eyes): we equip the virtual human with a Dynamic Scene-Aware Navigation, which continuously perceives changes in the surrounding environment and adaptively predicts the next waypoint. (2)Memory (human brain): we equip the virtual human with a Hierarchical Experience Memory, which stores and updates experiential data accumulated during training. This allows the model to leverage prior knowledge during inference for context-aware motion priming, thereby enhancing both motion quality and generalization. (3) Control (human body): we equip the virtual human with Human-Scene Interaction Diffusion Model, which generates high-fidelity interaction motions conditioned on multimodal inputs. To evaluate performance in dynamic scenes, we extend the existing static human-scene interaction datasets to construct a dynamic benchmark, Dyn-Scenes. We conduct extensive qualitative and quantitative experiments to validate Dyn-HSI, showing that our method consistently outperforms existing approaches and generates high-quality human-scene interaction motions in both static and dynamic settings.

Figures (7)

• Figure 1: Overview of Dyn-HSI. Dyn-HSI is an autoregressive motion diffusion model capable of iteratively generating human–scene interaction motions. It consists of three components: (1) dynamic scene-aware navigation, (2) hierarchical experience memory, and (3) HSI diffusion model.
• Figure 2: Architecture of Dynamic Scene-Aware Navigation.
• Figure 3: Construction instructions for the dynamic scene evaluation benchmark. The recorded data is shown on the right.
• Figure 4: User study results. The higher score indicates better performance.
• Figure 5: Visual results compared with existing methods. The left side provides a detailed description of the dynamic conditions within the scene. The green box zooms in on the details captured by our approach, while the red boxes highlight errors made by other methods. Green arrows indicate the goal positions.