Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models

Zhankai Ye, Bofan Li, Yukai Jin, Shuoqiu Li, Wei Wang, Yanfu Zhang, Shangqian Gao, Xin Liu

TL;DR

A novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other and achieves a 20% performance improvement over current state-of-the-art methods.

Abstract

Discrete motion tokenization has recently enabled Large Language Models (LLMs) to serve as versatile backbones for motion understanding and motion-language reasoning. However, existing pipelines typically decouple motion quantization from semantic embedding learning, linking them solely via token IDs. This approach fails to effectively align the intrinsic geometry of the motion space with the embedding space, thereby hindering the LLM's capacity for nuanced motion reasoning. We argue that alignment is most effective when both modalities share a unified geometric basis. Therefore, instead of forcing the LLM to reconstruct the complex geometry among motion tokens from scratch, we present a novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other. Specifically, we employ a decoder-only quantizer with Gumbel-Softmax for differentiable training and balanced codebook usage. To bridge the modalities, we use a sparse projection that maps motion codes into the LLM embedding space while preserving orthogonality. Finally, a two-stage orthonormal regularization schedule enforces soft constraints during tokenizer training and LLM fine-tuning to maintain geometric alignment without hindering semantic adaptation. Extensive experiments on HumanML3D demonstrate that our framework achieves a 20% performance improvement over current state-of-the-art methods, validating that a unified geometric basis effectively empowers the LLM for nuanced motion reasoning.

GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models

TL;DR

A novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other and achieves a 20% performance improvement over current state-of-the-art methods.

Abstract

Discrete motion tokenization has recently enabled Large Language Models (LLMs) to serve as versatile backbones for motion understanding and motion-language reasoning. However, existing pipelines typically decouple motion quantization from semantic embedding learning, linking them solely via token IDs. This approach fails to effectively align the intrinsic geometry of the motion space with the embedding space, thereby hindering the LLM's capacity for nuanced motion reasoning. We argue that alignment is most effective when both modalities share a unified geometric basis. Therefore, instead of forcing the LLM to reconstruct the complex geometry among motion tokens from scratch, we present a novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other. Specifically, we employ a decoder-only quantizer with Gumbel-Softmax for differentiable training and balanced codebook usage. To bridge the modalities, we use a sparse projection that maps motion codes into the LLM embedding space while preserving orthogonality. Finally, a two-stage orthonormal regularization schedule enforces soft constraints during tokenizer training and LLM fine-tuning to maintain geometric alignment without hindering semantic adaptation. Extensive experiments on HumanML3D demonstrate that our framework achieves a 20% performance improvement over current state-of-the-art methods, validating that a unified geometric basis effectively empowers the LLM for nuanced motion reasoning.
Paper Structure (27 sections, 13 equations, 3 figures, 5 tables)

This paper contains 27 sections, 13 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Approach
  4. Geometric Unification as Alignment
  5. Structure-Preserving Sparse Projection
  6. Decoder-Only Vector Quantization (DVQ)
  7. Two-Stage Orthonormal Regularization
  8. Experiment
  9. Experimental Settings
  10. Evaluation Metrics
  11. Codebook Distribution Analysis
  12. Performance on Motion Understanding
  13. Ablation Studies
  14. Effect of Orthogonal Loss Ratio.
  15. Impact of Different LLM Backbones.
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overall framework of GeoMotionGPT. Left: a DVQ-based motion tokenizer encodes an input motion $x$ into discrete codebook indices and reconstructs $\hat{x}$ via a decoder. Middle: we introduce an auto-alignment objective with orthogonality, encouraging the normalized codebook correlation (and its projected embedding counterpart) to approach the identity matrix. Right: the LLM vocabulary is extended with trainable motion-token embeddings while keeping the original text embeddings frozen, enabling multimodal motion-language training and inference.
  • Figure 2: Codebook utilization comparison between GeoMotionGPT and a conventional VQ-VAE. GeoMotionGPT achieves more effective code usage (less skewed heavy-tailed usage pattern).
  • Figure 3: Comparison between LLM Training with/without Ortho. Loss and without Sparse Projection