EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences

TL;DR

EmpathicStories++ introduces the first in-the-wild, longitudinal multimodal dataset for empathy toward personal experiences, collecting 53 hours of video, audio, and text from 41 participants in home deployments over a month, with self-reported empathy labels and psychometrics. The study defines a multimodal empathy-prediction task using a context window [t-k/2, ..., t+k/2] across video, audio, and text, and benchmarks multiple architectures including AMER, TFN, LSTM variants, EmpathicStoriesBART, and GPT-4. Key findings show strong GPT-4 performance in text-centric Story Share and improved multimodal performance in the Reflection context, illustrating context- and modality-dependent efficacy; ablations reveal the value of modality fusion in introspective tasks and the robustness of text-based prompts. This resource enables quantitative, longitudinal insights into human empathy and supports development of empathetic AI with real-world applicability, with the dataset openly released for further research on cognitive aspects of empathy.

Abstract

Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.

Figures (4)

• Figure 1: Data collection setup. The robot station houses a webcamera and microphone for video/audio data collection. A tablet displays stories read by participants, as well as sliders for self-rating empathy on a scale of 1-5.
• Figure 2: Basic dataset statistics. Video length and word count statistics of all participant sessions, as well as the distribution of self-rated empathy labels.
• Figure 3: Trait and state surveys. Participant demographic information, trait and state survey overviews show diversity across age, gender, personality type, and feelings of social connection and wellbeing over time.
• Figure 4: Story Topics: We visualize the embeddings (obtained with UMAP of ada-002 embeddings) of Story topics. Our deployment across the United States gives us a diverse set of meaningful personal stories.