Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors

Xiaoyan Cong, Zekun Li, Zhiyang Dou, Hongyu Li, Omid Taheri, Chuan Guo, Abhay Mittal, Sizhe An, Taku Komura, Wojciech Matusik, Michael J. Black, Srinath Sridhar

Abstract

Large-scale foundation models (LFMs) have recently made impressive progress in text-to-motion generation by learning strong generative priors from massive 3D human motion datasets and paired text descriptions. However, how to effectively and efficiently leverage such single-purpose motion LFMs, i.e., text-to-motion synthesis, in more diverse cross-modal and in-context motion generation downstream tasks remains largely unclear. Prior work typically adapts pretrained generative priors to individual downstream tasks in a task-specific manner. In contrast, our goal is to unlock such priors to support a broad spectrum of downstream motion generation tasks within a single unified framework. To bridge this gap, we present UMO, a simple yet general unified formulation that casts diverse downstream tasks into compositions of atomic per-frame operations, enabling in-context adaptation to unlock the generative priors of pretrained DiT-based motion LFMs. Specifically, UMO introduces three learnable frame-level meta-operation embeddings to specify per-frame intent and employs lightweight temporal fusion to inject in-context cues into the pretrained backbone, with negligible runtime overhead compared to the base model. With this design, UMO finetunes the pretrained model, originally limited to text-to-motion generation, to support diverse previously unsupported tasks, including temporal inpainting, text-guided motion editing, text-serialized geometric constraints, and multi-identity reaction generation. Experiments demonstrate that UMO consistently outperforms task-specific and training-free baselines across a wide range of benchmarks, despite using a single unified model. Code and model will be publicly available. Project Page: https://oliver-cong02.github.io/UMO.github.io/

UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors

Abstract

Large-scale foundation models (LFMs) have recently made impressive progress in text-to-motion generation by learning strong generative priors from massive 3D human motion datasets and paired text descriptions. However, how to effectively and efficiently leverage such single-purpose motion LFMs, i.e., text-to-motion synthesis, in more diverse cross-modal and in-context motion generation downstream tasks remains largely unclear. Prior work typically adapts pretrained generative priors to individual downstream tasks in a task-specific manner. In contrast, our goal is to unlock such priors to support a broad spectrum of downstream motion generation tasks within a single unified framework. To bridge this gap, we present UMO, a simple yet general unified formulation that casts diverse downstream tasks into compositions of atomic per-frame operations, enabling in-context adaptation to unlock the generative priors of pretrained DiT-based motion LFMs. Specifically, UMO introduces three learnable frame-level meta-operation embeddings to specify per-frame intent and employs lightweight temporal fusion to inject in-context cues into the pretrained backbone, with negligible runtime overhead compared to the base model. With this design, UMO finetunes the pretrained model, originally limited to text-to-motion generation, to support diverse previously unsupported tasks, including temporal inpainting, text-guided motion editing, text-serialized geometric constraints, and multi-identity reaction generation. Experiments demonstrate that UMO consistently outperforms task-specific and training-free baselines across a wide range of benchmarks, despite using a single unified model. Code and model will be publicly available. Project Page: https://oliver-cong02.github.io/UMO.github.io/
Paper Structure (26 sections, 2 equations, 15 figures, 12 tables)

This paper contains 26 sections, 2 equations, 15 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diverse Motion Generation Downstream Tasks
  4. Text-to-Motion Foundation Models
  5. Method
  6. Preliminary
  7. Unified In-Context Motion Generation Formulation
  8. In-Context Conditioning Architecture
  9. Experiment
  10. Experimental Setup
  11. Architecture Ablation
  12. In-Domain Task Adaptation
  13. Out-of-Domain Tasks Adaptation
  14. Stress Test: Ability Emergence
  15. Limitations
  16. ...and 11 more sections

Figures (15)

  • Figure 1: UMO casts diverse downstream tasks as compositions of our novel atomic per-frame operations, unlocking the generative priors of a pretrained text-to-motion LFM without any architectural modification in a broad spectrum of applications.
  • Figure 2: UMO achieves state-of-the-art performance and efficiency across diverse tasks. Latency, Traj. Err, $\texttt{[P]}$-MPJPE, and FID are inverted so that the outermost boundary is consistently optimal.
  • Figure 3: Architecture Alternatives for In-Context Motion Conditioning.
  • Figure 4: Ablation of In-context Conditioning Architectures in Keyframe Infilling.
  • Figure 5: Qualitative Comparison of Text-guided Motion Editing.
  • ...and 10 more figures