A Fuzzy-based Approach to Predict Human Interaction by Functional Near-Infrared Spectroscopy

TL;DR

This work introduces the Fuzzy Attention Layer, a fuzzy-logic–based self-attention kernel integrated into a Transformer Encoder to enhance interpretability when decoding fNIRS hyperscanning data from handholding social interactions. By replacing dot-product attention with a fuzzy firing-strength mechanism tied to Gaussian memberships, the model learns human-interpretable neural patterns while maintaining or improving predictive performance. The approach enables sample- and group-level analyses of neural activity, links attention rules to specific prefrontal cortex regions, and reveals increased inter-brain synchrony under handholding. The findings hold promise for advancing social neuroscience and psychologically informed AI, with potential extensions to EEG/fMRI and real-time neuroadaptive systems, and suggest directions for optimizing rule design and training efficiency.

Abstract

The paper introduces a Fuzzy-based Attention (Fuzzy Attention Layer) mechanism, a novel computational approach to enhance the interpretability and efficacy of neural models in psychological research. The proposed Fuzzy Attention Layer mechanism is integrated as a neural network layer within the Transformer Encoder model to facilitate the analysis of complex psychological phenomena through neural signals, such as those captured by functional Near-Infrared Spectroscopy (fNIRS). By leveraging fuzzy logic, the Fuzzy Attention Layer is capable of learning and identifying interpretable patterns of neural activity. This capability addresses a significant challenge when using Transformer: the lack of transparency in determining which specific brain activities most contribute to particular predictions. Our experimental results demonstrated on fNIRS data from subjects engaged in social interactions involving handholding reveal that the Fuzzy Attention Layer not only learns interpretable patterns of neural activity but also enhances model performance. Additionally, the learned patterns provide deeper insights into the neural correlates of interpersonal touch and emotional exchange. The application of our model shows promising potential in deciphering the subtle complexities of human social behaviors, thereby contributing significantly to the fields of social neuroscience and psychological AI.

Figures (10)

• Figure 1: Overall illustration of decoding handholding by hyperscanning fNIRS signals. A: A demonstration sample showing paired fNIRS signals. The brain image displays the prefrontal cortex's maximum concentration of HbO and HbR. The second plot illustrates how the average signal across channels for HbO and HbR changes over time. B: The main structure of our proposed model. Add&Norm: A residual connection followed by layer normalization. C: The center, which is identified by the Fuzzy Attention Layer. D: The structure of the Fuzzy Attention Layer. E: The firing strength of the demonstration sample.
• Figure 2: A: The T-values, which illustrate the differences between handholding (HH) and non-handholding (NHH) conditions across 10 different rules, are compared within the channel-first structure. B: The T-values, which illustrate the differences between HH and NHH conditions across 10 rules, are compared within the time-first structure. C: The centre prototype for rule #5.D: The placement of fNIRS sensors (sources and detectors) and channels. E: Statistical analysis results showing the (de-)similarity between the latent variables of two individuals. ***:p<0.001
• Figure 3: A: The 3 random-selected examples of fuzzy attention score. B: The random-selected dot attention score
• Figure 4: The stimulus diagram of the experiment for two datasets. A: Picture Recognition Dataset and B: Picture Rating Dataset. This figure illustrates how data were collected under the handholding (HH) and non-handholding (NHH) conditions.
• Figure 5: HRF, modelled by the standard SPM method. The figure illustrates the response over time to a stimulus of 1-second duration starting at 0 seconds. The response is depicted as a continuous curve, showcasing the typical biphasic pattern of the HRF, with an initial rise peaking shortly after stimulus onset followed by an undershoot. The shaded area represents the stimulus's duration, highlighting the neural activation's temporal context.