Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot

Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, Erik Cambria

TL;DR

The paper addresses the limitation of text-only empathetic response generation by introducing EmpathyEar, an open-source avatar-based multimodal empathetic chatbot. It integrates a multimodal encoding frontend (ImageBind), an LLM core (ChatGLM3), and dedicated speech and talking-face generators (StyleTTS2 and EAT) to produce synchronized multimodal outputs. A chain-of-thought based meta-response mechanism (emotion, scene context, content, agent profile) guides cross-modal generation, further enhanced by emotion-aware instruction-tuning and augmentation on emotion datasets. Evaluation shows superior performance versus non-LLM and multimodal baselines in automatic and human assessments, underscoring potential for richer, emotionally resonant interactions in diverse scenarios.

Abstract

This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access.

EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot

TL;DR

The paper addresses the limitation of text-only empathetic response generation by introducing EmpathyEar, an open-source avatar-based multimodal empathetic chatbot. It integrates a multimodal encoding frontend (ImageBind), an LLM core (ChatGLM3), and dedicated speech and talking-face generators (StyleTTS2 and EAT) to produce synchronized multimodal outputs. A chain-of-thought based meta-response mechanism (emotion, scene context, content, agent profile) guides cross-modal generation, further enhanced by emotion-aware instruction-tuning and augmentation on emotion datasets. Evaluation shows superior performance versus non-LLM and multimodal baselines in automatic and human assessments, underscoring potential for richer, emotionally resonant interactions in diverse scenarios.

Abstract

This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access.
Paper Structure (28 sections, 8 figures, 2 tables)

This paper contains 28 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Workflow
  4. Implementation Specification
  5. EmpathyEar Architecture
  6. Multimodal Encoding Module.
  7. Core LLM Reasoning Module.
  8. Speech & Talking-face Generation Module.
  9. CoT-based Meta-response Generation
  10. Emotion-aware Instruction-Tuning
  11. Encoder-LLM Alignment Learning.
  12. Meta-response Instruction-Tuning.
  13. Emotion-aware Enhancement Training.
  14. Use Scenario and Demonstration
  15. Application Scenario
  16. ...and 13 more sections

Figures (8)

  • Figure 1: The architecture of EmpathyEar, supporting avatar-based multimodal empathetic response generation.
  • Figure 2: Workflow of the EmpathyEar system.
  • Figure 3: Screenshot of the EmpathyEar providing companion service by chatting with an elder.
  • Figure 4: Screenshot of the dialogue between the user and EmpathyEar for psychological assistance.
  • Figure 5: Human evaluation in seven different aspects: Q1) accuracy of emotion recognition, Q2) fluency of the language, Q3) rationality of the analysis, Q4) clarity of speech, Q5) emotional consistency of speech, Q6) clarity of video facial features, and Q7) emotional consistency of video expressions.
  • ...and 3 more figures