Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Digital Avatars: Framework Development and Their Evaluation

Timothy Rupprecht, Sung-En Chang, Yushu Wu, Lei Lu, Enfu Nan, Chih-hsiang Li, Caiyue Lai, Zhimin Li, Zhijun Hu, Yumei He, David Kaeli, Yanzhi Wang

TL;DR

This paper tackles the challenge of creating anthropomorphic AI-driven avatars that exhibit humor, authenticity, and favorable perception. It introduces a show don't tell prompting strategy and an end-to-end, architecture-agnostic avatar framework that renders LLM outputs into real-time audio-video interactions. Evaluation is conducted via Crowd Vote, an adaptation of Crowd Score, enabling quantitative comparison of avatar responses on humor, authenticity, and favorability. Results show that avatars powered by the proposed prompting strategy outperform baselines and even real-world personas, demonstrated with Donald Trump and Joe Biden avatars. The work delivers both a practical avatar pipeline and a scalable evaluation methodology with implications for deploying engaging digital characters in interactive settings.

Abstract

We present a novel prompting strategy for artificial intelligence driven digital avatars. To better quantify how our prompting strategy affects anthropomorphic features like humor, authenticity, and favorability we present Crowd Vote - an adaptation of Crowd Score that allows for judges to elect a large language model (LLM) candidate over competitors answering the same or similar prompts. To visualize the responses of our LLM, and the effectiveness of our prompting strategy we propose an end-to-end framework for creating high-fidelity artificial intelligence (AI) driven digital avatars. This pipeline effectively captures an individual's essence for interaction and our streaming algorithm delivers a high-quality digital avatar with real-time audio-video streaming from server to mobile device. Both our visualization tool, and our Crowd Vote metrics demonstrate our AI driven digital avatars have state-of-the-art humor, authenticity, and favorability outperforming all competitors and baselines. In the case of our Donald Trump and Joe Biden avatars, their authenticity and favorability are rated higher than even their real-world equivalents.

Digital Avatars: Framework Development and Their Evaluation

TL;DR

This paper tackles the challenge of creating anthropomorphic AI-driven avatars that exhibit humor, authenticity, and favorable perception. It introduces a show don't tell prompting strategy and an end-to-end, architecture-agnostic avatar framework that renders LLM outputs into real-time audio-video interactions. Evaluation is conducted via Crowd Vote, an adaptation of Crowd Score, enabling quantitative comparison of avatar responses on humor, authenticity, and favorability. Results show that avatars powered by the proposed prompting strategy outperform baselines and even real-world personas, demonstrated with Donald Trump and Joe Biden avatars. The work delivers both a practical avatar pipeline and a scalable evaluation methodology with implications for deploying engaging digital characters in interactive settings.

Abstract

We present a novel prompting strategy for artificial intelligence driven digital avatars. To better quantify how our prompting strategy affects anthropomorphic features like humor, authenticity, and favorability we present Crowd Vote - an adaptation of Crowd Score that allows for judges to elect a large language model (LLM) candidate over competitors answering the same or similar prompts. To visualize the responses of our LLM, and the effectiveness of our prompting strategy we propose an end-to-end framework for creating high-fidelity artificial intelligence (AI) driven digital avatars. This pipeline effectively captures an individual's essence for interaction and our streaming algorithm delivers a high-quality digital avatar with real-time audio-video streaming from server to mobile device. Both our visualization tool, and our Crowd Vote metrics demonstrate our AI driven digital avatars have state-of-the-art humor, authenticity, and favorability outperforming all competitors and baselines. In the case of our Donald Trump and Joe Biden avatars, their authenticity and favorability are rated higher than even their real-world equivalents.
Paper Structure (14 sections, 2 figures, 1 table)

This paper contains 14 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Approach
  3. Prompt Mechanism
  4. An End-to-End Digital Avatar Framework
  5. Speech-to-Text and Text-to-Speech
  6. Talking Face Synthesis
  7. Video Selection Algorithm
  8. Evaluation Metrics
  9. Crowd Score
  10. Introducing Crowd Vote
  11. Experiments
  12. Interest and Humor
  13. Authenticity and Favorability
  14. Conclusion

Figures (2)

  • Figure 1: Proposed framework agnostic to specific architecture.
  • Figure 2: Each of our avatars with show don't tell prompting is selected by the judges as most authentic and most favorable when compared to a baseline avatar without show don't tell prompting, an avatar from character.ai, and even the real-world persona.