EEG-Based Analysis of Brain Responses in Multi-Modal Human-Robot Interaction: Modulating Engagement

TL;DR

Observations on neural responses during interaction indicate that the proposed multi-modal approach can effectively enhance user engagement, which is critical for improving outcomes.

Abstract

User engagement, cognitive participation, and motivation during task execution in physical human-robot interaction are crucial for motor learning. These factors are especially important in contexts like robotic rehabilitation, where neuroplasticity is targeted. However, traditional robotic rehabilitation systems often face challenges in maintaining user engagement, leading to unpredictable therapeutic outcomes. To address this issue, various techniques, such as assist-as-needed controllers, have been developed to prevent user slacking and encourage active participation. In this paper, we introduce a new direction through a novel multi-modal robotic interaction designed to enhance user engagement by synergistically integrating visual, motor, cognitive, and auditory (speech recognition) tasks into a single, comprehensive activity. To assess engagement quantitatively, we compared multiple electroencephalography (EEG) biomarkers between this multi-modal protocol and a traditional motor-only protocol. Fifteen healthy adult participants completed 100 trials of each task type. Our findings revealed that EEG biomarkers, particularly relative alpha power, showed statistically significant improvements in engagement during the multi-modal task compared to the motor-only task. Moreover, while engagement decreased over time in the motor-only task, the multi-modal protocol maintained consistent engagement, suggesting that users could remain engaged for longer therapy sessions. Our observations on neural responses during interaction indicate that the proposed multi-modal approach can effectively enhance user engagement, which is critical for improving outcomes. This is the first time that objective neural response highlights the benefit of a comprehensive robotic intervention combining motor, cognitive, and auditory functions in healthy subjects.

Figures (7)

• Figure 1: Overview of experiment design and processing pipeline, showing (A) the experimental setup, (B) the EEG processing steps and brain regions, and (C) an example of sEMG epoching.
• Figure 2: Mean relative PSD heatmaps of the brain during the planning phase of the Motor-Only and Matching tasks. Results are shown for (A) delta band, (B) theta band, and (C) alpha band.
• Figure 3: Boxplots showing relative PSD for different regions of the brain during the planning phase of the Motor-Only and Matching tasks. Results are shown for (A) delta band, (B) theta band, and (C) alpha band. Significant differences, per the paired t-test, are indicated with asterisks. *: $p<0.05$, **: $p<0.01$, ***: $p<0.001$, and ****: $p<0.0001$.
• Figure 4: Mean relative PSD heatmaps of the brain during the movement phase of the Motor-Only and Matching tasks. Results are shown for (A) delta band, (B) theta band, and (C) alpha band.
• Figure 5: Boxplots showing relative PSD for different regions of the brain during the movement phase of the Motor-Only and Matching tasks. Results are shown for (A) delta band, (B) theta band, and (C) alpha band. Significant differences, per the paired t-test, are indicated with asterisks. *: $p<0.05$, **: $p<0.01$, ***: $p<0.001$, and ****: $p<0.0001$.