THERADIA WoZ: An Ecological Corpus for Appraisal-based Affect Research in Healthcare

TL;DR

This work addresses the need for ecological, multimodal affect data in healthcare to support AI-assisted cognitive training. It introduces THERADIA WoZ, a French-language audiovisual corpus collected from healthy aging adults and MCI patients during Wizard-of-Oz–driven CCT sessions, with full transcripts and appraisal-based annotations across four dimensions and 23 affect labels. The authors provide data collection protocols, annotation guidelines, a detailed corpus analysis, and baseline automatic recognition results using both hand-crafted and self-supervised features across audio, text, and video modalities, including fusion strategies. The corpus fills a critical gap in healthcare affective computing and offers a valuable benchmark for industry and academia to develop affect-aware home-based AI assistants for CCT, with potential implications for personalized therapy and elder care.

Abstract

We present THERADIA WoZ, an ecological corpus designed for audiovisual research on affect in healthcare. Two groups of senior individuals, consisting of 52 healthy participants and 9 individuals with Mild Cognitive Impairment (MCI), performed Computerised Cognitive Training (CCT) exercises while receiving support from a virtual assistant, tele-operated by a human in the role of a Wizard-of-Oz (WoZ). The audiovisual expressions produced by the participants were fully transcribed, and partially annotated based on dimensions derived from recent models of the appraisal theories, including novelty, intrinsic pleasantness, goal conduciveness, and coping. Additionally, the annotations included 23 affective labels drew from the literature of achievement affects. We present the protocols used for the data collection, transcription, and annotation, along with a detailed analysis of the annotated dimensions and labels. Baseline methods and results for their automatic prediction are also presented. The corpus aims to serve as a valuable resource for researchers in affective computing, and is made available to both industry and academia.

Figures (6)

• Figure 1: Head movements, gaze, speech, and articulation of a remote operator were captured in real-time to drive a virtual assistant as a Wizard-of-Oz Tarpin-Bernard21-TDT.
• Figure 2: Office of the experimenter operating the virtual assistant. The dialog was fully scripted and overlaid onto the participant's video (A). The participant's screen (B) was only visible during the exercise completion, and the virtual assistant was visible to the participant only outside the exercise completion. The virtual assistant's rendering was consistently displayed (C) to monitor any tracking issue.
• Figure 3: Example of the ANNOT web-base platform for time-continuous annotation of the intrinsic pleasantness dimension.
• Figure 4: Frequency (logarithmic scale) of the annotated labels according to the agreement of at least one, two or three annotators among six.
• Figure 5: Heatmap of the PCC between affective labels and summary values of appraisal dimensions aggregated across all annotators. The sign (+) or (-) after each label denotes whether it is positive or negative, respectively. Dimensions are as follows: AR: arousal, CP: coping, GC: goal conduciveness, IP: intrinsic pleasantness, NV: novelty. The correlations shown represent the average between annotators for whom the correlation was significant (p < 0.05) after Bonferroni correction and of identical direction. Positive or negative direction is shown in blue or red, respectively. A correlation present for a single annotator was considered null.