Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SentiAvatar: Towards Expressive and Interactive Digital Humans

Chuhao Jin, Rui Zhang, Qingzhe Gao, Haoyu Shi, Dayu Wu, Yichen Jiang, Yihan Wu, Ruihua Song

Abstract

We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.

SentiAvatar: Towards Expressive and Interactive Digital Humans

Abstract

We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.

Paper Structure

This paper contains 48 sections, 13 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-driven Human Motion Generation
  4. Speech-driven Gesture and Expression Generation
  5. Motion Representation, Foundation Models, and Datasets
  6. Constructing SuSuInterActs Dataset
  7. Dataset Analysis.
  8. Dataset Comparison
  9. Method
  10. System Overview
  11. Discrete Representation
  12. LLM-based Semantic Planning
  13. Audio-conditioned Motion Infilling
  14. Facial Expression Generation
  15. Motion Foundation Model
  16. ...and 33 more sections

Figures (4)

  • Figure 1: SentiAvatar generates high-quality 3D human motion and expression, which are semantically aligned and frame-level synchronized. The same color indicates the same time step.
  • Figure 2: Overview of SuSuInterActs data pipeline. (1) Character design for a consistent persona. (2) LLM script generation with explicit motion/expression tags. (3) Multimodal capture of synchronized speech, full-body motion, and facial expressions.
  • Figure 3: Overview of SentiAvatar. (a) Multi-modal inputs are quantized into tokens via encoders. The body pathway employs a hierarchical generation: (b) LLM planner predicts sparse keyframe tokens to capture high-level dialogue content, followed by (c) an audio-aware Infill Transformer for dense, prosody-driven interpolation to ensure fine-grained temporal synchronization.
  • Figure 4: Qualitative comparison of generated motions across methods. Each row shows keyframe sequences for a given motion and speech. Texts and arrows of the same color indicate the same time step. The red arrow indicates an incorrect action.