MultiPhysio-HRC: Multimodal Physiological Signals Dataset for industrial Human-Robot Collaboration

TL;DR

The paper introduces MultiPhysio-HRC, a publicly available multimodal dataset for industrial Human-Robot Collaboration that combines EEG, ECG, EDA, RESP, EMG, facial action units, audio, and video collected in real-world HRC tasks, VR scenarios, and cognitive challenges. It provides rich ground-truth labels from validated questionnaires and ground-truth task conditions, enabling analysis of stress, cognitive load, and emotion. The two-day protocol includes baseline/rest, cognitive and VR tasks, followed by manual and robot-assisted battery disassembly, with a detailed data acquisition and processing pipeline. Baseline models indicate physiological signals are highly informative for stress and cognitive load estimation, highlighting the potential of multimodal fusion for human-aware robotics in industry. The dataset aims to accelerate research in affective computing and safe, human-centered automation in industrial settings.

Abstract

Human-robot collaboration (HRC) is a key focus of Industry 5.0, aiming to enhance worker productivity while ensuring well-being. The ability to perceive human psycho-physical states, such as stress and cognitive load, is crucial for adaptive and human-aware robotics. This paper introduces MultiPhysio-HRC, a multimodal dataset containing physiological, audio, and facial data collected during real-world HRC scenarios. The dataset includes electroencephalography (EEG), electrocardiography (ECG), electrodermal activity (EDA), respiration (RESP), electromyography (EMG), voice recordings, and facial action units. The dataset integrates controlled cognitive tasks, immersive virtual reality experiences, and industrial disassembly activities performed manually and with robotic assistance, to capture a holistic view of the participants' mental states. Rich ground truth annotations were obtained using validated psychological self-assessment questionnaires. Baseline models were evaluated for stress and cognitive load classification, demonstrating the dataset's potential for affective computing and human-aware robotics research. MultiPhysio-HRC is publicly available to support research in human-centered automation, workplace well-being, and intelligent robotic systems.

Figures (6)

• Figure 1: Data acquisition protocol.
• Figure 2: Displayed screen of each cognitive task: SCWT (top left), N-Back (bottom left), Arithmetic (top right), and Hanoi tower (bottom right).
• Figure 3: Experimental robotic cell setup. The multiple components of the disassembled battery can be seen placed on the table.
• Figure 4: Battery disassembly steps.
• Figure 5: Sample of the acquired physiological data. The participant signals are filtered and normalized (min-max).