Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing

Yang Xiao, Wang Lu, Jie Ji, Ruimeng Ye, Gen Li, Xiaolong Ma, Bo Hui

TL;DR

This paper constructs a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching and unveils the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments.

Abstract

The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with stimulus signals using Mean Squared Error (MSE), which focuses only on local point-wise alignment and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain signals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11\% in single-subject training and 3.81\% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available at https://github.com/NKUShaw/OT-Alignment4brain-to-image.

Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing

TL;DR

This paper constructs a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching and unveils the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments.

Abstract

The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with stimulus signals using Mean Squared Error (MSE), which focuses only on local point-wise alignment and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain signals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11\% in single-subject training and 3.81\% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available at https://github.com/NKUShaw/OT-Alignment4brain-to-image.

Paper Structure

This paper contains 18 sections, 20 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Brain-Image Alignment.
  4. Optimal Transport.
  5. Brodmann's Areas
  6. Problem Setup
  7. Preliminary
  8. Mean Squared Error (MSE)
  9. Method
  10. Framework
  11. Optimal Transport
  12. OT Loss and MSE Loss
  13. Experiment
  14. Experiment Setup
  15. Alignment Experiments
  16. ...and 3 more sections

Figures (7)

  • Figure 1: The left is MSE Heatmap and the right is OT Heatmap.
  • Figure 2: Framework: our OT Loss not only considers the point-wise alignment of MSE but also the global relationships.
  • Figure 3: Each row of brain images from upper to down is related to left ventral, right ventral, left lateral, and right lateral. Each column from left to right is related to Body, Face, Place, and Word region.
  • Figure 4: The regions' importance.
  • Figure 5: The Redundant and Synergistic of three metrics in different regions
  • ...and 2 more figures