Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars

Ziqiao Peng, Yi Chen, Yifeng Ma, Guozhen Zhang, Zhiyao Sun, Zixiang Zhou, Youliang Zhang, Zhengguang Zhou, Zhaoxin Fan, Hongyan Liu, Yuan Zhou, Qinglin Lu, Jun He

TL;DR

ActAvatar addresses the challenges of precise text-driven action control and temporal alignment in talking avatars by introducing Phase-Aware Cross-Attention for phase-wise temporal grounding, Progressive Audio-Visual Alignment to reduce modality interference, and a Two-Stage Training strategy to preserve audio-visual alignment and text-following. The method uses structured prompts generated by an MLLM, a diffusion-based image-to-video backbone, and a carefully staged training process to inject temporally anchored actions without forgetting lip synchronization. Extensive experiments across HDTF and Action Bench show ActAvatar achieving state-of-the-art action control and visual quality, with efficient inference. The work demonstrates the viability of text-driven, temporally precise action control in realistic talking avatars, reducing reliance on explicit pose signals.

Abstract

Despite significant advances in talking avatar generation, existing methods face critical challenges: insufficient text-following capability for diverse actions, lack of temporal alignment between actions and audio content, and dependency on additional control signals such as pose skeletons. We present ActAvatar, a framework that achieves phase-level precision in action control through textual guidance by capturing both action semantics and temporal context. Our approach introduces three core innovations: (1) Phase-Aware Cross-Attention (PACA), which decomposes prompts into a global base block and temporally-anchored phase blocks, enabling the model to concentrate on phase-relevant tokens for precise temporal-semantic alignment; (2) Progressive Audio-Visual Alignment, which aligns modality influence with the hierarchical feature learning process-early layers prioritize text for establishing action structure while deeper layers emphasize audio for refining lip movements, preventing modality interference; (3) A two-stage training strategy that first establishes robust audio-visual correspondence on diverse data, then injects action control through fine-tuning on structured annotations, maintaining both audio-visual alignment and the model's text-following capabilities. Extensive experiments demonstrate that ActAvatar significantly outperforms state-of-the-art methods in both action control and visual quality.

ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars

TL;DR

ActAvatar addresses the challenges of precise text-driven action control and temporal alignment in talking avatars by introducing Phase-Aware Cross-Attention for phase-wise temporal grounding, Progressive Audio-Visual Alignment to reduce modality interference, and a Two-Stage Training strategy to preserve audio-visual alignment and text-following. The method uses structured prompts generated by an MLLM, a diffusion-based image-to-video backbone, and a carefully staged training process to inject temporally anchored actions without forgetting lip synchronization. Extensive experiments across HDTF and Action Bench show ActAvatar achieving state-of-the-art action control and visual quality, with efficient inference. The work demonstrates the viability of text-driven, temporally precise action control in realistic talking avatars, reducing reliance on explicit pose signals.

Abstract

Despite significant advances in talking avatar generation, existing methods face critical challenges: insufficient text-following capability for diverse actions, lack of temporal alignment between actions and audio content, and dependency on additional control signals such as pose skeletons. We present ActAvatar, a framework that achieves phase-level precision in action control through textual guidance by capturing both action semantics and temporal context. Our approach introduces three core innovations: (1) Phase-Aware Cross-Attention (PACA), which decomposes prompts into a global base block and temporally-anchored phase blocks, enabling the model to concentrate on phase-relevant tokens for precise temporal-semantic alignment; (2) Progressive Audio-Visual Alignment, which aligns modality influence with the hierarchical feature learning process-early layers prioritize text for establishing action structure while deeper layers emphasize audio for refining lip movements, preventing modality interference; (3) A two-stage training strategy that first establishes robust audio-visual correspondence on diverse data, then injects action control through fine-tuning on structured annotations, maintaining both audio-visual alignment and the model's text-following capabilities. Extensive experiments demonstrate that ActAvatar significantly outperforms state-of-the-art methods in both action control and visual quality.
Paper Structure (23 sections, 11 equations, 4 figures, 4 tables)

This paper contains 23 sections, 11 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Generation Models
  4. Talking Avatar Generation
  5. Method
  6. Overview
  7. Phase-Aware Cross-Attention
  8. Hierarchical Prompt Decomposition
  9. Phase-Conditioned Attention Dynamics
  10. Progressive Audio-Visual Alignment
  11. Two-Stage Training Strategy
  12. Stage 1: Audio Adapter Training and Extraction
  13. Stage 2: Temporally-Aware Action Control
  14. Experiments
  15. Experimental Setup
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Overview of ActAvatar. Given an audio input and reference image, ActAvatar generates temporally-controlled action videos guided by structured prompts automatically generated from MLLM.
  • Figure 2: Qualitative comparison with state-of-the-art methods. ActAvatar accurately executes phase-specific actions with correct timing and clear hand articulation, while competing methods show temporal misalignment, vague motions, or degraded hand quality.
  • Figure 3: Cross-attention phase focus at layer 5 (top) and layer 20 (bottom). Deeper layers show sharper phase separation.
  • Figure 4: Ablation study on PACA. Top: Without PACA, the avatar remains static throughout the sequence. Bottom: With PACA, the avatar naturally walks forward.