Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

Morteza Bodaghi, Majid Hosseini, Raju Gottumukkala

TL;DR

This study addresses stress detection by combining biometric signals and facial landmarks through an intermediate fusion network. It systematically evaluates six manifold-learning dimensionality reduction methods applied per modality, identifying Multidimensional Scaling (MDS) as the top performer with 96% accuracy under LOSO-CV, albeit at high preprocessing cost. The results reveal clear accuracy-cost trade-offs across unimodal, early fusion, and intermediate fusion setups, with PCA offering fast baselines and MDS delivering superior multimodal performance. Additional gains arise from data balancing and an extra non-linear 1D-CNN layer after fusion, highlighting practical strategies for implementing efficient, real-world stress-detection systems.

Abstract

Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learning by simplifying data and making the features more amenable to processing and analysis, thereby reducing computational complexity. This paper introduces an intermediate multimodal fusion network with manifold learning-based dimensionality reduction. The multimodal network generates independent representations from biometric signals and facial landmarks through 1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN layer, followed by a fully connected dense layer. We compared various dimensionality reduction techniques for different variations of unimodal and multimodal networks. We observe that the intermediate-level fusion with the Multi-Dimensional Scaling (MDS) manifold method showed promising results with an accuracy of 96.00\% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV) paradigm over other dimensional reduction methods. MDS had the highest computational cost among manifold learning methods. However, while outperforming other networks, it managed to reduce the computational cost of the proposed networks by 25\% when compared to six well-known conventional feature selection methods used in the preprocessing step.

A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

TL;DR

This study addresses stress detection by combining biometric signals and facial landmarks through an intermediate fusion network. It systematically evaluates six manifold-learning dimensionality reduction methods applied per modality, identifying Multidimensional Scaling (MDS) as the top performer with 96% accuracy under LOSO-CV, albeit at high preprocessing cost. The results reveal clear accuracy-cost trade-offs across unimodal, early fusion, and intermediate fusion setups, with PCA offering fast baselines and MDS delivering superior multimodal performance. Additional gains arise from data balancing and an extra non-linear 1D-CNN layer after fusion, highlighting practical strategies for implementing efficient, real-world stress-detection systems.

Abstract

Multimodal deep learning methods capture synergistic features from multiple modalities and have the potential to improve accuracy for stress detection compared to unimodal methods. However, this accuracy gain typically comes from high computational cost due to the high-dimensional feature spaces, especially for intermediate fusion. Dimensionality reduction is one way to optimize multimodal learning by simplifying data and making the features more amenable to processing and analysis, thereby reducing computational complexity. This paper introduces an intermediate multimodal fusion network with manifold learning-based dimensionality reduction. The multimodal network generates independent representations from biometric signals and facial landmarks through 1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN layer, followed by a fully connected dense layer. We compared various dimensionality reduction techniques for different variations of unimodal and multimodal networks. We observe that the intermediate-level fusion with the Multi-Dimensional Scaling (MDS) manifold method showed promising results with an accuracy of 96.00\% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV) paradigm over other dimensional reduction methods. MDS had the highest computational cost among manifold learning methods. However, while outperforming other networks, it managed to reduce the computational cost of the proposed networks by 25\% when compared to six well-known conventional feature selection methods used in the preprocessing step.
Paper Structure (18 sections, 9 figures, 5 tables)

This paper contains 18 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Intermediate Level Fusion Network
  4. Manifold learning based Dimensionality Reduction
  5. Locally Linear Embedding (LLE)
  6. Spectral Embedding (SE)
  7. Multi-Dimensional Scaling (MDS)
  8. Isometric Mapping (ISO)
  9. t-Distributed Stochastic Neighbor Embedding (t-SNE)
  10. Principal Component Analysis (PCA)
  11. Results and Analysis
  12. Unimodal Networks
  13. Multimodal Early Level Fusion Network
  14. Multimodal Intermediate Level Fusion Network
  15. Computational Cost
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Intermediate Level Fusion Network
  • Figure 2: Biometric and Landmarks data visualization using LLE
  • Figure 3: Biometric and Landmarks data visualization using SE
  • Figure 4: Biometric and Landmarks data visualization using MDS
  • Figure 5: Biometric and Landmarks data visualization using ISO
  • ...and 4 more figures