Towards Affect-Adaptive Human-Robot Interaction: A Protocol for Multimodal Dataset Collection on Social Anxiety
Vesna Poprcova, Iulia Lefter, Matthias Wieser, Martijn Warnier, Frances Brazier
TL;DR
This paper tackles the scarcity of multimodal datasets for social anxiety in human-robot interaction by proposing a detailed protocol to collect a synchronized dataset. It combines audio, video, and physiological signals from at least 70 participants during 10-minute Wizard-of-Oz interactions with the Furhat robot, complemented by online and in-lab questionnaires and rich contextual metadata. The protocol features two robot communication styles, a mixed factorial design, and robust ethical considerations to enable comprehensive multimodal affective modeling and fusion. By releasing a public dataset and promoting affect-adaptive HRI, the work aims to advance realistic robot-driven support for individuals with social anxiety and inform future studies in group settings.
Abstract
Social anxiety is a prevalent condition that affects interpersonal interactions and social functioning. Recent advances in artificial intelligence and social robotics offer new opportunities to examine social anxiety in the human-robot interaction context. Accurate detection of affective states and behaviours associated with social anxiety requires multimodal datasets, where each signal modality provides complementary insights into its manifestations. However, such datasets remain scarce, limiting progress in both research and applications. To address this, this paper presents a protocol for multimodal dataset collection designed to reflect social anxiety in a human-robot interaction context. The dataset will consist of synchronised audio, video, and physiological recordings acquired from at least 70 participants, grouped according to their level of social anxiety, as they engage in approximately 10-minute interactive Wizard-of-Oz role-play scenarios with the Furhat social robot under controlled experimental conditions. In addition to multimodal data, the dataset will be enriched with contextual data providing deeper insight into individual variability in social anxiety responses. This work can contribute to research on affect-adaptive human-robot interaction by providing support for robust multimodal detection of social anxiety.