Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SOSControl: Enhancing Human Motion Generation through Saliency-Aware Symbolic Orientation and Timing Control

Ho Yin Au, Junkun Jiang, Jie Chen

TL;DR

SOSControl addresses the lack of fine-grained control in text-to-motion by introducing the Salient Orientation Symbolic (SOS) script, a programmable symbolic interface for body-part orientations and motion timing at keyframes. It combines an automatic SOS extraction pipeline using temporally constrained agglomerative clustering with a Saliency-based Masking Scheme to produce sparse, interpretable scripts and a data augmentation strategy. The SOSControl framework integrates SOS signals into diffusion-based motion generation via ControlNet adaptation, gradient-based iterative optimization, and an ACTOR-PAE–driven periodic latent space to ensure smooth, natural motion outputs. Across HumanML3D and BABEL, SOSControl demonstrates improved controllability and robust alignment with user-specified orientation and timing constraints, enabling more interpretable and actionable control for animation, robotics, and interactive AI applications.

Abstract

Traditional text-to-motion frameworks often lack precise control, and existing approaches based on joint keyframe locations provide only positional guidance, making it challenging and unintuitive to specify body part orientations and motion timing. To address these limitations, we introduce the Salient Orientation Symbolic (SOS) script, a programmable symbolic framework for specifying body part orientations and motion timing at keyframes. We further propose an automatic SOS extraction pipeline that employs temporally-constrained agglomerative clustering for frame saliency detection and a Saliency-based Masking Scheme (SMS) to generate sparse, interpretable SOS scripts directly from motion data. Moreover, we present the SOSControl framework, which treats the available orientation symbols in the sparse SOS script as salient and prioritizes satisfying these constraints during motion generation. By incorporating SMS-based data augmentation and gradient-based iterative optimization, the framework enhances alignment with user-specified constraints. Additionally, it employs a ControlNet-based ACTOR-PAE Decoder to ensure smooth and natural motion outputs. Extensive experiments demonstrate that the SOS extraction pipeline generates human-interpretable scripts with symbolic annotations at salient keyframes, while the SOSControl framework outperforms existing baselines in motion quality, controllability, and generalizability with respect to motion timing and body part orientation control.

SOSControl: Enhancing Human Motion Generation through Saliency-Aware Symbolic Orientation and Timing Control

TL;DR

SOSControl addresses the lack of fine-grained control in text-to-motion by introducing the Salient Orientation Symbolic (SOS) script, a programmable symbolic interface for body-part orientations and motion timing at keyframes. It combines an automatic SOS extraction pipeline using temporally constrained agglomerative clustering with a Saliency-based Masking Scheme to produce sparse, interpretable scripts and a data augmentation strategy. The SOSControl framework integrates SOS signals into diffusion-based motion generation via ControlNet adaptation, gradient-based iterative optimization, and an ACTOR-PAE–driven periodic latent space to ensure smooth, natural motion outputs. Across HumanML3D and BABEL, SOSControl demonstrates improved controllability and robust alignment with user-specified orientation and timing constraints, enabling more interpretable and actionable control for animation, robotics, and interactive AI applications.

Abstract

Traditional text-to-motion frameworks often lack precise control, and existing approaches based on joint keyframe locations provide only positional guidance, making it challenging and unintuitive to specify body part orientations and motion timing. To address these limitations, we introduce the Salient Orientation Symbolic (SOS) script, a programmable symbolic framework for specifying body part orientations and motion timing at keyframes. We further propose an automatic SOS extraction pipeline that employs temporally-constrained agglomerative clustering for frame saliency detection and a Saliency-based Masking Scheme (SMS) to generate sparse, interpretable SOS scripts directly from motion data. Moreover, we present the SOSControl framework, which treats the available orientation symbols in the sparse SOS script as salient and prioritizes satisfying these constraints during motion generation. By incorporating SMS-based data augmentation and gradient-based iterative optimization, the framework enhances alignment with user-specified constraints. Additionally, it employs a ControlNet-based ACTOR-PAE Decoder to ensure smooth and natural motion outputs. Extensive experiments demonstrate that the SOS extraction pipeline generates human-interpretable scripts with symbolic annotations at salient keyframes, while the SOSControl framework outperforms existing baselines in motion quality, controllability, and generalizability with respect to motion timing and body part orientation control.
Paper Structure (48 sections, 8 equations, 6 figures, 6 tables)

This paper contains 48 sections, 8 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary
  4. Motion Data Representation
  5. Kinematic Feature Extraction
  6. Salient Orientation Symbols
  7. Kinematic Feature Extraction
  8. Hierarchical Temporal Saliency Detection
  9. Spatial Feature Quantization
  10. Saliency-based Masking Scheme
  11. Motion Periodic Latent Diffusion with Saliency Orientation Script
  12. Basic Periodic Latent Diffusion
  13. ACTOR-PAE and Periodic Latent Space.
  14. Periodic Latent Diffusion.
  15. Injecting SOS Guidance to the Diffusion Process.
  16. ...and 33 more sections

Figures (6)

  • Figure 1: Salient Orientation Script (SOS) Illustration: (a) Body Part and (b) Root Orientation Symbols specify keyframe states for effective motion control. (c) SOS example as a staff highlights its programmable interface potential.
  • Figure 2: Overview of the SOS extraction pipeline: (a) extract per-frame orientation features, then (b) quantize them according to symbol categories from Fig. \ref{['fig:symbol_illust']}. Agglomerative clustering is applied in (c) to derive the per-frame orientation feature into frame saliency shown in (d). Symbols selected from frames with saliency above a threshold compose the final SOS.
  • Figure 3: Overview of the SOSControl pipeline: (a) Start by obtaining a semantic condition and a Symbolic Orientation Script (SOS) from user input, (b) perform motion periodic latent diffusion, and (c) decode the resulting motion latent back to motion. Both (b) and (c) utilize ControlNet to incorporate the SOS condition into the trained models and apply iterative optimization in (e) to refine the model output, ensuring better alignment with the input SOS conditions. To improve model adaptability to diverse user-provided SOS scripts, (d) perform SMS-based mask sampling to generate SOS at varying levels of granularity, enabling the trained models to handle different saliency variations more robustly.
  • Figure 4: Visualization of SOS-conditioned motion generation results with and without SMS training data processing. (a) presents the motion sequences from left to right, while (b) provides a detailed view of the range between frames 24 and 33.
  • Figure 5: Visualization of SOS-conditioned motion generation results with and without test-time iterative optimization. (a) presents the motion sequences from left to right, while (b) provides a detailed view of the range between frames 9 and 18.
  • ...and 1 more figures