Transformer with Controlled Attention for Synchronous Motion Captioning

TL;DR

The paper tackles synchronous motion captioning by framing text generation as time-aligned with human motion sequences. It introduces a Transformer variant with masked self- and cross-attention, plus monotonic alignment losses and a learnable center $m_t$ to enforce progressive, frame-focused generation. Key contributions include window-based masking (with $Γ_i$ and $\\gamma_t$), a single-layer Transformer design for interpretable attention, and alignment losses that improve synchronization on KIT-ML and HumanML3D, accompanied by qualitative attention visualizations. This approach advances explainable, time-synchronous language generation for motion data and has implications for sign language alignment and unsupervised action localization.

Abstract

In this paper, we address a challenging task, synchronous motion captioning, that aim to generate a language description synchronized with human motion sequences. This task pertains to numerous applications, such as aligned sign language transcription, unsupervised action segmentation and temporal grounding. Our method introduces mechanisms to control self- and cross-attention distributions of the Transformer, allowing interpretability and time-aligned text generation. We achieve this through masking strategies and structuring losses that push the model to maximize attention only on the most important frames contributing to the generation of a motion word. These constraints aim to prevent undesired mixing of information in attention maps and to provide a monotonic attention distribution across tokens. Thus, the cross attentions of tokens are used for progressive text generation in synchronization with human motion sequences. We demonstrate the superior performance of our approach through evaluation on the two available benchmark datasets, KIT-ML and HumanML3D. As visual evaluation is essential for this task, we provide a comprehensive set of animated visual illustrations in the code repository: https://github.com/rd20karim/Synch-Transformer.

Figures (18)

• Figure 1: Overview of general proposed framework with relevant details.
• Figure 2: Cross attention map of compositional motions with corresponding frame range of each action (D=r=10). Across multiple examples, we observe that the attention distribution of motion words consistently falls within the indicated motion range for each specific action.
• Figure 3: Frozen motion with 4 keyframes of higher attention corresponding to the language segment.
• Figure 4: Decomposition of motions and associated descriptions (Animations in the code repository, other visualizations in Supp.\ref{['supp:more_vizu']}).
• Figure 5: Multi-Layer Transformer: Compared to Figure \ref{['fig:walk_turn_walk']} attention distributions, here, are uninformative about action times, attention weights (for 'turns', 'walks back down') are not aligned with action times (same observation for different samples).