Towards Multimodal Emotional Support Conversation Systems

TL;DR

This work introduces the Multimodal Emotional Support Conversation dataset, a first-of-its-kind resource enriched with comprehensive annotations across text, audio, and video modalities, and proposes a general Sequential Multimodal Emotional Support framework (SMES) grounded in Therapeutic Skills Theory.

Abstract

The integration of conversational artificial intelligence (AI) into mental health care promises a new horizon for therapist-client interactions, aiming to closely emulate the depth and nuance of human conversations. Despite the potential, the current landscape of conversational AI is markedly limited by its reliance on single-modal data, constraining the systems' ability to empathize and provide effective emotional support. This limitation stems from a paucity of resources that encapsulate the multimodal nature of human communication essential for therapeutic counseling. To address this gap, we introduce the Multimodal Emotional Support Conversation (MESC) dataset, a first-of-its-kind resource enriched with comprehensive annotations across text, audio, and video modalities. This dataset captures the intricate interplay of user emotions, system strategies, system emotion, and system responses, setting a new precedent in the field. Leveraging the MESC dataset, we propose a general Sequential Multimodal Emotional Support framework (SMES) grounded in Therapeutic Skills Theory. Tailored for multimodal dialogue systems, the SMES framework incorporates an LLM-based reasoning model that sequentially generates user emotion recognition, system strategy prediction, system emotion prediction, and response generation. Our rigorous evaluations demonstrate that this framework significantly enhances the capability of AI systems to mimic therapist behaviors with heightened empathy and strategic responsiveness. By integrating multimodal data in this innovative manner, we bridge the critical gap between emotion recognition and emotional support, marking a significant advancement in conversational AI for mental health support.

Figures (6)

• Figure 1: An example chat from the MESC dataset, where the client's and therapist's emotions are highlighted in bold red. Therapeutic strategies used by the therapist, informed by the client's emotions, are marked in blue, showcasing how multimodal information supports emotional engagement.
• Figure 2: The proportion of scenarios of MESC.
• Figure 3: The distribution of strategies at different conversation progress.
• Figure 4: The SMES framework uses multimodality information as inputs to improve mental health support. It employs Video-Llama to extract emotional cues from video and audio, then processes them through the LLM-based Reasoning model to sequentially generate four emotional-related task results.
• Figure 5: The LLM-based reasoning modal consolidates all emotion-related sub-tasks into a sequence-to-sequence generation framework, optimizing them in an end-to-end manner.