Dense Motion Captioning

TL;DR

This work introduces Dense Motion Captioning (DMC), a task to localize and describe multiple temporally-bound actions in 3D human motion. To support this, it presents CompMo, a large-scale dataset of 60,000 long motion sequences with dense, timestamped action captions, and DEMO, a baseline model that fuses a lightweight motion adapter with a Large Language Model via a two-stage training regime. DEMO demonstrates strong gains over prior methods on CompMo and adapted benchmarks, establishing a robust baseline for temporally grounded 3D motion understanding and language grounding. The results highlight the feasibility of open-ended, text-based descriptions for complex motion streams and point to future work in spatio-temporal reasoning and long-horizon behavioral modeling.

Abstract

Recent advances in 3D human motion and language integration have primarily focused on text-to-motion generation, leaving the task of motion understanding relatively unexplored. We introduce Dense Motion Captioning, a novel task that aims to temporally localize and caption actions within 3D human motion sequences. Current datasets fall short in providing detailed temporal annotations and predominantly consist of short sequences featuring few actions. To overcome these limitations, we present the Complex Motion Dataset (CompMo), the first large-scale dataset featuring richly annotated, complex motion sequences with precise temporal boundaries. Built through a carefully designed data generation pipeline, CompMo includes 60,000 motion sequences, each composed of multiple actions ranging from at least two to ten, accurately annotated with their temporal extents. We further present DEMO, a model that integrates a large language model with a simple motion adapter, trained to generate dense, temporally grounded captions. Our experiments show that DEMO substantially outperforms existing methods on CompMo as well as on adapted benchmarks, establishing a robust baseline for future research in 3D motion understanding and captioning.

Figures (5)

• Figure 1: Dense Motion Captioning (DMC). We present DMC, a task that localizes and generates detailed segment-level captions with accurate temporal boundaries in 3D human motion sequences. To support this task, we construct CompMo, the first large-scale 3D motion-language dataset providing dense captions for multiple temporal segments within each motion sequence. Each sequence contains between 2 and 10 atomic actions, and every action is annotated with precise timestamps and a descriptive caption.
• Figure 2: Single Motion Captioning performance divided by simple and complex motion sequences. We report the single motion captioning performance of state-of-the-art motion-language models on the simple and complex subsets of HumanML3D Guo_HumanML3D's test set, as defined in Sec. \ref{['sec:preliminary']}.
• Figure 3: DEMO overview : Given a motion sequence $m$, our method encodes it with the motion adapter $\Phi_{\textbf{W},\gamma}$, which maps it into the language embedding space of the LLM $f_\phi$. Using the resulting motion embeddings and a textual instruction $x_{inst}$, the model generates dense captions with temporal boundaries. Training is conducted in two stages. Here, denotes the subset of parameters being trained.
• Figure 4: Qualitative Results. We show two motion sequence examples from the CompMo dataset, along with the ground truth annotations (GT) and the dense captions predicted by our DEMO and UniMotion. For each sequence, the top rows show the temporal intervals of the input motion divided according to the GT and the two model predictions, with the corresponding captions listed below. Predicted captions that align with the GT are highlighted in the same color and connected with arrows to indicate the alignment.
• Figure A.1: Overview of CompMo generation pipeline We illustrate the three steps of the data generation pipeline, as detailed in \ref{['sec:data']}.