Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Streamlined Facial Data Collection based on Utterance and Emotional Data for Human-to-Avatar Reconstruction

Seoyoung Kang, Seokhwan Yang, Hail Song, Boram Yoon, Jinwook Kim, Kangsoo Kim, Woontack Woo

TL;DR

The paper tackles the problem of data-intensive photorealistic avatar reconstruction by proposing a data-efficient approach tailored to conversational contexts. It introduces two key inputs—utterance data and emotional data—and tests them in a two-phase design: Phase 1 identifies effective utterance modalities and emotion capture methods, while Phase 2 evaluates user-perceived realism, naturalness, and telepresence across three data-collection scenarios. The findings show that a targeted combination of spontaneous speech and direct emotional expression can match the perceptual quality of extensive datasets while reducing data and training time, with equivalence demonstrated between the streamlined C2 condition and the extensive C3 baseline. The work provides practical guidelines for real-time avatar generation in AR/VR telepresence, suggesting that focused data captures suffice for high-quality, efficient, and engaging conversational avatars.

Abstract

This study explores a streamlined facial data collection method for conversational contexts, addressing the limitations of existing approaches that often require extensive datasets and prioritize technical metrics over user perception and experience. We systematically investigate which facial expression data are essential for reconstructing photorealistic avatars and how they can be captured efficiently. Our research employs a two-phase methodology to identify efficient facial data collection strategies and evaluate their effectiveness. In the first phase, we conduct facial data acquisition and evaluate reconstruction performance using utterance data and emotional data. In the second phase, we carry out a comprehensive user evaluation comparing three progressive conditions: utterance only, utterance and emotional data, and a control condition involving extensive data. Findings from 24 participants engaged in simulated face-to-face conversations reveal that targeted utterance and emotional data achieve comparable levels of perceived realism, naturalness, and telepresence, while reducing training time and data usage when compared to the extensive data collection approach. These results demonstrate that targeted data inputs can enable efficient avatar face reconstruction, offering practical guidelines for real-time applications such as AR/VR telepresence and highlighting the trade-off between data quantity and perceived quality.

Streamlined Facial Data Collection based on Utterance and Emotional Data for Human-to-Avatar Reconstruction

TL;DR

The paper tackles the problem of data-intensive photorealistic avatar reconstruction by proposing a data-efficient approach tailored to conversational contexts. It introduces two key inputs—utterance data and emotional data—and tests them in a two-phase design: Phase 1 identifies effective utterance modalities and emotion capture methods, while Phase 2 evaluates user-perceived realism, naturalness, and telepresence across three data-collection scenarios. The findings show that a targeted combination of spontaneous speech and direct emotional expression can match the perceptual quality of extensive datasets while reducing data and training time, with equivalence demonstrated between the streamlined C2 condition and the extensive C3 baseline. The work provides practical guidelines for real-time avatar generation in AR/VR telepresence, suggesting that focused data captures suffice for high-quality, efficient, and engaging conversational avatars.

Abstract

This study explores a streamlined facial data collection method for conversational contexts, addressing the limitations of existing approaches that often require extensive datasets and prioritize technical metrics over user perception and experience. We systematically investigate which facial expression data are essential for reconstructing photorealistic avatars and how they can be captured efficiently. Our research employs a two-phase methodology to identify efficient facial data collection strategies and evaluate their effectiveness. In the first phase, we conduct facial data acquisition and evaluate reconstruction performance using utterance data and emotional data. In the second phase, we carry out a comprehensive user evaluation comparing three progressive conditions: utterance only, utterance and emotional data, and a control condition involving extensive data. Findings from 24 participants engaged in simulated face-to-face conversations reveal that targeted utterance and emotional data achieve comparable levels of perceived realism, naturalness, and telepresence, while reducing training time and data usage when compared to the extensive data collection approach. These results demonstrate that targeted data inputs can enable efficient avatar face reconstruction, offering practical guidelines for real-time applications such as AR/VR telepresence and highlighting the trade-off between data quantity and perceived quality.
Paper Structure (40 sections, 3 figures, 3 tables)

This paper contains 40 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Avatars for Telepresence in AR/VR
  4. Photorealistic Avatars and Face Reconstruction
  5. Methodology
  6. Facial Data Acquisition and Evaluation
  7. Utterance Data Acquisition
  8. Emotional Data Acquisition
  9. Comprehensive Experiment
  10. Apparatus
  11. Procedure
  12. Measures
  13. Performance Evaluation
  14. User Evaluation
  15. Study I: Facial Data Acquisition and Performance Evaluation
  16. ...and 25 more sections

Figures (3)

  • Figure 1: (Left) Data collection setup showing the front-facing participant from the front view and (Right) the side view.
  • Figure 2: Unity view of the avatars used in the user evaluation. (Top) Male Avatar, (Bottom) Female Avatar. (Left) Utterance only, (Middle) Utterance and Emotion, (Right) Extensive Data Combination.
  • Figure 3: (Top) Results of Likert scale ratings (1: strongly disagree – 7: strongly agree) for (a) Realism, (b) Naturalness, (c) Telepresence, and (d) Synchrony of Communication cues. (Bottom) Equivalence testing results comparing C2 and C3 for (e) Realism, (f) Naturalness, and (g) Telepresence.