Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing

Jitesh Joshi, Sos S. Agaian, Youngjun Cho

TL;DR

The Factorized Self-Attention Module (FSAM) is proposed, which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization and develops FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames.

Abstract

Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing

TL;DR

The Factorized Self-Attention Module (FSAM) is proposed, which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization and develops FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames.

Abstract

Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

Paper Structure

This paper contains 34 sections, 10 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Attention Mechanisms in Vision
  4. Attention Mechanisms in rPPG Methods
  5. Method
  6. Primer: Nonnegative Matrix Factorization
  7. Factorized Self-Attention Module (FSAM)
  8. Deployment of FSAM in 3D-CNN and 2D-CNN Architectures
  9. FactorizePhys Architecture:
  10. Adaptation of FSAM for 2D-CNN Architecture:
  11. Experiments
  12. Results and Discussion
  13. Ablation Study:
  14. FactorizePhys vs. State-of-the-Art:
  15. Computation Cost and Latency:
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Formulation of Nonnegative Matrix Factorization (NMF)
  • Figure 2: Factorized Self-Attention Module (FSAM) illustrated for a 3D-CNN architecture for rPPG estimation.
  • Figure 3: (A) Proposed FactorizePhys with FSAM; (B) FSAM Adapted for EfficientPhys liu2023EfficientPhys
  • Figure 4: (A) Cumulative cross-dataset performance (MAE) v/s latency† plot. The size of the sphere corresponds to the number of model parameters; (B) Visualization of learned spatial-temporal features from the base 3D-CNN model trained without and with FSAM; † System specs: Ubuntu 22.04 OS, NVIDIA GeForce RTX 3070 Laptop GPU, Intel® Core™ i7-10870H CPU @ 2.20GHz, 16 GB RAM.
  • Figure 5: Cross-dataset performance comparison between SOTA and the proposed method reported with cumulative evaluation metrics, their standard error (SE) and latency
  • ...and 4 more figures