Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses

Haidong Gu, Nathan Gaw, Yinan Wang, Chancellor Johnstone, Christine Beauchene, Sophia Yuditskaya, Hrishikesh Rao, Chun-An Chou

TL;DR

A hierarchical heterogeneous graph generative network (H2G2-Net) that automatically learns a graph structure without domain knowledge, as well as a powerful representation on the hierarchical heterogeneous graph in an end-to-end fashion is proposed.

Abstract

Discovering human cognitive and emotional states using multi-modal physiological signals draws attention across various research applications. Physiological responses of the human body are influenced by human cognition and commonly used to analyze cognitive states. From a network science perspective, the interactions of these heterogeneous physiological modalities in a graph structure may provide insightful information to support prediction of cognitive states. However, there is no clue to derive exact connectivity between heterogeneous modalities and there exists a hierarchical structure of sub-modalities. Existing graph neural networks are designed to learn on non-hierarchical homogeneous graphs with pre-defined graph structures; they failed to learn from hierarchical, multi-modal physiological data without a pre-defined graph structure. To this end, we propose a hierarchical heterogeneous graph generative network (H2G2-Net) that automatically learns a graph structure without domain knowledge, as well as a powerful representation on the hierarchical heterogeneous graph in an end-to-end fashion. We validate the proposed method on the CogPilot dataset that consists of multi-modal physiological signals. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art GNNs by 5%-20% in prediction accuracy.

H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses

TL;DR

A hierarchical heterogeneous graph generative network (H2G2-Net) that automatically learns a graph structure without domain knowledge, as well as a powerful representation on the hierarchical heterogeneous graph in an end-to-end fashion is proposed.

Abstract

Discovering human cognitive and emotional states using multi-modal physiological signals draws attention across various research applications. Physiological responses of the human body are influenced by human cognition and commonly used to analyze cognitive states. From a network science perspective, the interactions of these heterogeneous physiological modalities in a graph structure may provide insightful information to support prediction of cognitive states. However, there is no clue to derive exact connectivity between heterogeneous modalities and there exists a hierarchical structure of sub-modalities. Existing graph neural networks are designed to learn on non-hierarchical homogeneous graphs with pre-defined graph structures; they failed to learn from hierarchical, multi-modal physiological data without a pre-defined graph structure. To this end, we propose a hierarchical heterogeneous graph generative network (H2G2-Net) that automatically learns a graph structure without domain knowledge, as well as a powerful representation on the hierarchical heterogeneous graph in an end-to-end fashion. We validate the proposed method on the CogPilot dataset that consists of multi-modal physiological signals. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art GNNs by 5%-20% in prediction accuracy.
Paper Structure (10 sections, 4 equations, 3 figures, 1 table)

This paper contains 10 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Dataset
  3. Method
  4. Sub-modality level
  5. Modality level
  6. Experiments
  7. Results analysis
  8. Ablation studies
  9. Interpretability of the H2G2-Net
  10. Conclusion

Figures (3)

  • Figure 1: Data collection via physiological measurements recorded during immersive VR flight training simulations.
  • Figure 2: The architecture of the hierarchical heterogeneous graph generative network (H2G2-Net). Node denotes feature vector of modality or sub-modality. Node size measures amount of information. Node color represents modality. The feature vector of sub-modality is raw physiological signal. Edge denotes information flow among modalities or sub-modalities. Edge color and width indicate importance of information flow.
  • Figure 3: The sequence of adjacency matrices (i.e., dynamic graph structures) learned by the H2G2-Net.