Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

Mohammadreza Jamalifard, Yaxiong Lei, Parasto Azizinezhad, Javier Fumanal-Idocin, Javier Andreu-Perez

Abstract

We propose a neuro-symbolic architecture that learns four interpretable physiological concepts, oculomotor dynamics, gaze stability, prefrontal hemodynamics, and multimodal, from eye-tracking and neural hemodynamics, functional near-infrared spectroscopy, (fNIRS) windows using attention-based encoders, and combines them with differentiable approximate reasoning rules using learned weights and soft thresholds, to address both rigid hand-crafted rules and the lack of subject-level alignment diagnostics. We apply this system to fatigue classification from multimodal physiological signals, a domain that requires models that are accurate and interpretable, with internal reasoning that can be inspected for safety-critical use. In leave-one-subject-out evaluation on 18 participants (560 samples), the method achieves 72.1% +/- 12.3% accuracy, comparable to tuned baselines while exposing concept activations and rule firing strengths. Ablations indicate gains from participant-specific calibration (+5.2 pp), a modest drop without the fNIRS concept (-1.2 pp), and slightly better performance with Lukasiewicz operators than product (+0.9 pp). We also introduce concept fidelity, an offline per-subject audit metric from held-out labels, which correlates strongly with per-subject accuracy (r=0.843, p < 0.0001).

A Neuro-Symbolic System for Interpretable Multimodal Physiological Signals Integration in Human Fatigue Detection

Abstract

We propose a neuro-symbolic architecture that learns four interpretable physiological concepts, oculomotor dynamics, gaze stability, prefrontal hemodynamics, and multimodal, from eye-tracking and neural hemodynamics, functional near-infrared spectroscopy, (fNIRS) windows using attention-based encoders, and combines them with differentiable approximate reasoning rules using learned weights and soft thresholds, to address both rigid hand-crafted rules and the lack of subject-level alignment diagnostics. We apply this system to fatigue classification from multimodal physiological signals, a domain that requires models that are accurate and interpretable, with internal reasoning that can be inspected for safety-critical use. In leave-one-subject-out evaluation on 18 participants (560 samples), the method achieves 72.1% +/- 12.3% accuracy, comparable to tuned baselines while exposing concept activations and rule firing strengths. Ablations indicate gains from participant-specific calibration (+5.2 pp), a modest drop without the fNIRS concept (-1.2 pp), and slightly better performance with Lukasiewicz operators than product (+0.9 pp). We also introduce concept fidelity, an offline per-subject audit metric from held-out labels, which correlates strongly with per-subject accuracy (r=0.843, p < 0.0001).

Paper Structure

This paper contains 30 sections, 6 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Physiological Fatigue Detection
  4. Neuro-Symbolic AI and Approximate Reasoning
  5. Uncertainty and Reliability in Classification
  6. Experimental Setup
  7. Data Collection
  8. Baseline Methods
  9. Methods
  10. Problem Formulation
  11. Participant-Aware Normalization
  12. Concept Extraction with Learned Attention
  13. Differentiable Approximate Reasoning with Learned Weights
  14. Training Objective
  15. Concept Fidelity
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Neuro-symbolic architecture: participant-normalized features $\to$ concept bottleneck (C1--C4) $\to$ differentiable Logic Rules $\to$ fatigue score.
  • Figure 2: Case study (P007): fatigue prediction and labels (top), concept activations $C_1$--$C_4$ (middle), and rule $R_2$ firing (bottom); dashed line marks the Alert$\rightarrow$Fatigued transition.