A neuroergonomics model to evaluating nuclear power plants operators' performance under heat stress driven by ECG time-frequency spectrums and fNIRS prefrontal cortex network: a CNN-GAT fusion model

TL;DR

The paper tackles real-time evaluation of nuclear power plant operators under heat stress by integrating ECG time-frequency spectrums and fNIRS-based PFC network data. It introduces a CNN-GAT fusion framework, where ECG spectrograms are processed by CNNs and fNIRS connectivity is modeled as an undirected brain-network using Graph Attention Networks, providing a 3-class performance classification. The approach achieves a peak micro-AUC of 0.90 and accuracy of 81.82% (Model D), demonstrating that leveraging brain-network structure with multi-physiological signals improves robustness and generalization. The work offers a practical neuroergonomics tool for Industry 5.0, enabling proactive monitoring and intervention to maintain operator safety and performance in extreme environments.

Abstract

Operators experience complicated physiological and psychological states when exposed to extreme heat stress, which can impair cognitive function and decrease performance significantly, ultimately leading to severe secondary disasters. Therefore, there is an urgent need for a feasible technique to identify their abnormal states to enhance the reliability of human-cybernetics systems. With the advancement of deep learning in physiological modeling, a model for evaluating operators' performance driven by electrocardiogram (ECG) and functional near-infrared spectroscopy (fNIRS) was proposed, demonstrating high ecological validity. The model fused a convolutional neural network (CNN) backbone and a graph attention network (GAT) backbone to extract discriminative features from ECG time-frequency spectrums and fNIRS prefrontal cortex (PFC) network respectively with deeper neuroscience domain knowledge, and eventually achieved 0.90 AUC. Results supported that handcrafted features extracted by specialized neuroscience methods can alleviate overfitting. Inspired by the small-world nature of the brain network, the fNIRS PFC network was organized as an undirected graph and embedded by GAT. It is proven to perform better in information aggregation and delivery compared to a simple non-linear transformation. The model provides a potential neuroergonomics application for evaluating the human state in vital human-cybernetics systems under industry 5.0 scenarios.

Figures (11)

• Figure 1: Overview of the processing flow in this paper. ECG time-frequency spectrums were obtained by STFT, and PFC networks were obtained with activation betas as node features and functional connectivity as edge features.
• Figure 2: The experimental flow in each scenario.
• Figure 3: Some instances of the time-frequency spectrums resized to 64*64 pixels which are inputs to CNNs. Each row contains spectrums of different tasks from different subjects and is arranged in categories 1, 2, and 3 from top to bottom.
• Figure 4: Illustration of fNIRS data process flow.
• Figure 5: Illustration of our proposed models. Model A: a. model B: a+b. model C: a+b+c. model D: a+b+d. a: a three-classification model with a sequence of CNN convs based on spectral maps. b: 1*20 ECG vector was concatenated to the output features after the fully connected layer 1. c: 1*136 fNIRS vector was concatenated to the output features after the fully connected layer 1. d: 1*136 fNIRS vector was organized as a PFC network and embedded by GAT backbone to obtain GAT-learned vector. 1*32 GAT-learned vector was concatenated to the output features after the fully connected layer 1.