Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenes

Zichen Geng, Zeeshan Hayder, Wei Liu, Ajmal Mian

TL;DR

UniHM tackles the challenge of scene-aware human motion generation by unifying Text-to-Motion and Text-to-HOI under a diffusion-based framework conditioned on text, spatial waypoints, and scene context. It introduces a mixed-motion representation that combines continuous $6$DoF motion with discrete local motion tokens, and a Look-Up Free Quantization VAE (LFQ-VAE) that uses a codebook-free latent space to improve reconstruction and generation. The approach is complemented by augmenting the Lingo dataset with HumanML3D annotations to enhance scene-conditioned learning. Empirically, UniHM achieves competitive performance on the OMOMO HOI benchmark and on HumanML3D, with notable gains from waypoint-guided generation and LFQ-VAE, and demonstrates robust scene-aware motion synthesis with reduced collisions and improved interactions.

Abstract

Human motion synthesis in complex scenes presents a fundamental challenge, extending beyond conventional Text-to-Motion tasks by requiring the integration of diverse modalities such as static environments, movable objects, natural language prompts, and spatial waypoints. Existing language-conditioned motion models often struggle with scene-aware motion generation due to limitations in motion tokenization, which leads to information loss and fails to capture the continuous, context-dependent nature of 3D human movement. To address these issues, we propose UniHM, a unified motion language model that leverages diffusion-based generation for synthesizing scene-aware human motion. UniHM is the first framework to support both Text-to-Motion and Text-to-Human-Object Interaction (HOI) in complex 3D scenes. Our approach introduces three key contributions: (1) a mixed-motion representation that fuses continuous 6DoF motion with discrete local motion tokens to improve motion realism; (2) a novel Look-Up-Free Quantization VAE (LFQ-VAE) that surpasses traditional VQ-VAEs in both reconstruction accuracy and generative performance; and (3) an enriched version of the Lingo dataset augmented with HumanML3D annotations, providing stronger supervision for scene-specific motion learning. Experimental results demonstrate that UniHM achieves comparative performance on the OMOMO benchmark for text-to-HOI synthesis and yields competitive results on HumanML3D for general text-conditioned motion generation.

UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenes

TL;DR

UniHM tackles the challenge of scene-aware human motion generation by unifying Text-to-Motion and Text-to-HOI under a diffusion-based framework conditioned on text, spatial waypoints, and scene context. It introduces a mixed-motion representation that combines continuous DoF motion with discrete local motion tokens, and a Look-Up Free Quantization VAE (LFQ-VAE) that uses a codebook-free latent space to improve reconstruction and generation. The approach is complemented by augmenting the Lingo dataset with HumanML3D annotations to enhance scene-conditioned learning. Empirically, UniHM achieves competitive performance on the OMOMO HOI benchmark and on HumanML3D, with notable gains from waypoint-guided generation and LFQ-VAE, and demonstrates robust scene-aware motion synthesis with reduced collisions and improved interactions.

Abstract

Human motion synthesis in complex scenes presents a fundamental challenge, extending beyond conventional Text-to-Motion tasks by requiring the integration of diverse modalities such as static environments, movable objects, natural language prompts, and spatial waypoints. Existing language-conditioned motion models often struggle with scene-aware motion generation due to limitations in motion tokenization, which leads to information loss and fails to capture the continuous, context-dependent nature of 3D human movement. To address these issues, we propose UniHM, a unified motion language model that leverages diffusion-based generation for synthesizing scene-aware human motion. UniHM is the first framework to support both Text-to-Motion and Text-to-Human-Object Interaction (HOI) in complex 3D scenes. Our approach introduces three key contributions: (1) a mixed-motion representation that fuses continuous 6DoF motion with discrete local motion tokens to improve motion realism; (2) a novel Look-Up-Free Quantization VAE (LFQ-VAE) that surpasses traditional VQ-VAEs in both reconstruction accuracy and generative performance; and (3) an enriched version of the Lingo dataset augmented with HumanML3D annotations, providing stronger supervision for scene-specific motion learning. Experimental results demonstrate that UniHM achieves comparative performance on the OMOMO benchmark for text-to-HOI synthesis and yields competitive results on HumanML3D for general text-conditioned motion generation.
Paper Structure (19 sections, 11 equations, 5 figures, 5 tables)

This paper contains 19 sections, 11 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Motion
  4. Text-to-HOI
  5. Human Scene Interaction
  6. Method
  7. UniHM Dataset
  8. Look-Up Free Quantization VAE
  9. Sparse Waypoint Generation
  10. Full Motion Sequence Generation
  11. Experiments
  12. Evaluation Metrics and Baselines
  13. Implementation Details
  14. Quantative Results
  15. Qualatative Comparison
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Text-to-Motion sequences (left) and Text-to-HOI sequences (right) generated by our approach.
  • Figure 2: UniHM architecture and Look-up Free Quantization Variational Autoencoder.
  • Figure 3: Qualitative Comparison. The first two rows are for Text-to-HOI, and the 3-rd & the 4-th rows are for Text-to-Motion
  • Figure 4: Motions generated in the scene with/without waypoint guidance
  • Figure 5: Comparison of rFID and gFID for VQ-VAE and LFQ-VAE across different tasks.