Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation

Saif Punjwani, Larry Heck

TL;DR

Allo-AVA, a large-scale dataset specifically designed for text and audio-driven avatar gesture animation in an allocentric (third person point-of-view) context, enables the development and evaluation of more natural, context-aware avatar animation models, potentially transforming applications ranging from virtual reality to digital assistants.

Abstract

The scarcity of high-quality, multimodal training data severely hinders the creation of lifelike avatar animations for conversational AI in virtual environments. Existing datasets often lack the intricate synchronization between speech, facial expressions, and body movements that characterize natural human communication. To address this critical gap, we introduce Allo-AVA, a large-scale dataset specifically designed for text and audio-driven avatar gesture animation in an allocentric (third person point-of-view) context. Allo-AVA consists of $\sim$1,250 hours of diverse video content, complete with audio, transcripts, and extracted keypoints. Allo-AVA uniquely maps these keypoints to precise timestamps, enabling accurate replication of human movements (body and facial gestures) in synchronization with speech. This comprehensive resource enables the development and evaluation of more natural, context-aware avatar animation models, potentially transforming applications ranging from virtual reality to digital assistants.

Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation

TL;DR

Allo-AVA, a large-scale dataset specifically designed for text and audio-driven avatar gesture animation in an allocentric (third person point-of-view) context, enables the development and evaluation of more natural, context-aware avatar animation models, potentially transforming applications ranging from virtual reality to digital assistants.

Abstract

The scarcity of high-quality, multimodal training data severely hinders the creation of lifelike avatar animations for conversational AI in virtual environments. Existing datasets often lack the intricate synchronization between speech, facial expressions, and body movements that characterize natural human communication. To address this critical gap, we introduce Allo-AVA, a large-scale dataset specifically designed for text and audio-driven avatar gesture animation in an allocentric (third person point-of-view) context. Allo-AVA consists of 1,250 hours of diverse video content, complete with audio, transcripts, and extracted keypoints. Allo-AVA uniquely maps these keypoints to precise timestamps, enabling accurate replication of human movements (body and facial gestures) in synchronization with speech. This comprehensive resource enables the development and evaluation of more natural, context-aware avatar animation models, potentially transforming applications ranging from virtual reality to digital assistants.

Paper Structure

This paper contains 56 sections, 3 equations, 17 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Dataset Creation
  3. Data Collection Process
  4. Data Processing Pipeline
  5. Video and Audio Extraction
  6. Transcription
  7. Keypoint Extraction
  8. OpenPose
  9. MediaPipe
  10. Combining OpenPose and MediaPipe
  11. Keypoint Coordinate System
  12. Dataset Statistics and Structure
  13. Dataset Analysis
  14. 3D Keypoint Distribution
  15. Speech Rate vs. Movement Intensity
  16. ...and 41 more sections

Figures (17)

  • Figure 1: Allo-AVA data processing pipeline
  • Figure 2: Keypoint extraction process combining OpenPose and MediaPipe
  • Figure 3: 3D distribution of keypoints extracted from the Allo-AVA dataset. X and Y axes represent normalized frame coordinates, while the Z-axis (and color) represents normalized depth.
  • Figure 4: Scatter plot of speech rate versus movement intensity, with correlation statistics.
  • Figure 5: Temporal heatmap of movement intensity across different body parts.
  • ...and 12 more figures