PhysioFormer: Integrating Multimodal Physiological Signals and Symbolic Regression for Explainable Affective State Prediction

TL;DR

PhysioFormer model integrates individual attributes and multimodal physiological data to address inter-individual variability, enhancing its reliability and generalization across different individuals, and includes an explainability model that uses symbolic regression to extract laws linking physiological signals to affective states, increasing transparency and explainability.

Abstract

Most affective computing tasks still rely heavily on traditional methods, with few deep learning models applied, particularly in multimodal signal processing. Given the importance of stress monitoring for mental health, developing a highly reliable and accurate affective computing model is essential. In this context, we propose a novel model, for affective state prediction using physiological signals. PhysioFormer model integrates individual attributes and multimodal physiological data to address interindividual variability, enhancing its reliability and generalization across different individuals. By incorporating feature embedding and affective representation modules, PhysioFormer model captures dynamic changes in time-series data and multimodal signal features, significantly improving accuracy. The model also includes an explainability model that uses symbolic regression to extract laws linking physiological signals to affective states, increasing transparency and explainability. Experiments conducted on the Wrist and Chest subsets of the WESAD dataset confirmed the model's superior performance, achieving over 99% accuracy, outperforming existing SOTA models. Sensitivity and ablation experiments further demonstrated PhysioFormer's reliability, validating the contribution of its individual components. The integration of symbolic regression not only enhanced model explainability but also highlighted the complex relationships between physiological signals and affective states. Future work will focus on optimizing the model for larger datasets and real-time applications, particularly in more complex environments. Additionally, further exploration of physiological signals and environmental factors will help build a more comprehensive affective computing system, advancing its use in health monitoring and psychological intervention.

Figures (27)

• Figure 1: PhysioFormer model architecture consists of three submodules: the Feature Embedding Module, Affective Representation Module, and Prediction Module. The Feature Embedding Module encodes physiological data, the Affective Representation Module builds on these encoded features, and the Prediction Module forecasts the individual's current affective state. The Explanation model analyzes data within the trained model, generating feature importance scores and selecting key features, followed by symbolic regression to derive formulas that explain and quantify the influence of physiological indicators on affective states.
• Figure 3: The figure shows the process of Explanation model. First, the input data undergoes feature extraction and affective state representation through the Feature Embedding and Affective Representation modules. The processed features and state information are used to symbolic distillation, where feature importance scores generated by the PhysioFormer model are used to select key features. Next, symbolic regression is employed to generate the predicted value $\tilde{e}$, which is compared with the model's predicted value $e$ thereby extracting and generating symbolic laws for the physiological indicators.
• Figure 4: Convergence trends of the PhysioFormer model across datasets by splitting the WESAD dataset through windows of different sizes. Although convergence trends are shown on all datasets, there are differences in the speed of convergence and the magnitude of losses.
• Figure 5: The role of feature embedding in affective computation tasks on both datasets, with the results presented in terms of ACC. $No\_att\_ACC$ represents the performance of the PhysioFormer model without using the feature embedding module, while $Win30\_ACC$ indicates the performance of the PhysioFormer model after applying the feature embedding module.
• Figure 6: The role of individual attributes features in affective computation tasks on both datasets, with the results presented in terms of ACC. $No\_pf\_ACC$ represents the performance of the PhysioFormer model without combining individual attributes features, while $Win30\_ACC$ indicates the performance of the PhysioFormer model after combining individual attributes features.