Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning

Weihang You, Hanqi Jiang, Yi Pan, Junhao Chen, Tianming Liu, Fei Dou

TL;DR

NeuroAlign tackles the challenge of fine-grained fMRI-video alignment under HRF delays, noise, and hierarchical visual coding by introducing a brain-inspired framework with three core components. Neural-Temporal Contrastive Learning (NTCL) captures temporal dynamics and delays to establish global semantic alignment, while Vector Quantization (VQ) provides robust, discrete representations for fine-grained matching across multiple cortical scales. The DynaSyncMM-EMA mechanism ensures balanced, variance-aware multi-modal fusion by synchronizing codebook updates and weighting modalities by their noise levels. Empirical results on HCP and CC2017 show NeuroAlign achieving 1.4–1.8× improvements across six retrieval directions and substantial ablation-supported gains, indicating a new paradigm for understanding visual cognitive mechanisms and enabling robust cross-modal retrieval with potential clinical applicability.

Abstract

Understanding neural responses to visual stimuli remains challenging due to the inherent complexity of brain representations and the modality gap between neural data and visual inputs. Existing methods, mainly based on reducing neural decoding to generation tasks or simple correlations, fail to reflect the hierarchical and temporal processes of visual processing in the brain. To address these limitations, we present NeuroAlign, a novel framework for fine-grained fMRI-video alignment inspired by the hierarchical organization of the human visual system. Our framework implements a two-stage mechanism that mirrors biological visual pathways: global semantic understanding through Neural-Temporal Contrastive Learning (NTCL) and fine-grained pattern matching through enhanced vector quantization. NTCL explicitly models temporal dynamics through bidirectional prediction between modalities, while our DynaSyncMM-EMA approach enables dynamic multi-modal fusion with adaptive weighting. Experiments demonstrate that NeuroAlign significantly outperforms existing methods in cross-modal retrieval tasks, establishing a new paradigm for understanding visual cognitive mechanisms.

Achieving Fine-grained Cross-modal Understanding through Brain-inspired Hierarchical Representation Learning

TL;DR

NeuroAlign tackles the challenge of fine-grained fMRI-video alignment under HRF delays, noise, and hierarchical visual coding by introducing a brain-inspired framework with three core components. Neural-Temporal Contrastive Learning (NTCL) captures temporal dynamics and delays to establish global semantic alignment, while Vector Quantization (VQ) provides robust, discrete representations for fine-grained matching across multiple cortical scales. The DynaSyncMM-EMA mechanism ensures balanced, variance-aware multi-modal fusion by synchronizing codebook updates and weighting modalities by their noise levels. Empirical results on HCP and CC2017 show NeuroAlign achieving 1.4–1.8× improvements across six retrieval directions and substantial ablation-supported gains, indicating a new paradigm for understanding visual cognitive mechanisms and enabling robust cross-modal retrieval with potential clinical applicability.

Abstract

Understanding neural responses to visual stimuli remains challenging due to the inherent complexity of brain representations and the modality gap between neural data and visual inputs. Existing methods, mainly based on reducing neural decoding to generation tasks or simple correlations, fail to reflect the hierarchical and temporal processes of visual processing in the brain. To address these limitations, we present NeuroAlign, a novel framework for fine-grained fMRI-video alignment inspired by the hierarchical organization of the human visual system. Our framework implements a two-stage mechanism that mirrors biological visual pathways: global semantic understanding through Neural-Temporal Contrastive Learning (NTCL) and fine-grained pattern matching through enhanced vector quantization. NTCL explicitly models temporal dynamics through bidirectional prediction between modalities, while our DynaSyncMM-EMA approach enables dynamic multi-modal fusion with adaptive weighting. Experiments demonstrate that NeuroAlign significantly outperforms existing methods in cross-modal retrieval tasks, establishing a new paradigm for understanding visual cognitive mechanisms.
Paper Structure (11 sections, 8 equations, 3 figures, 2 tables)

This paper contains 11 sections, 8 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Framework Overview
  4. Neural-Temporal Contrastive Learning
  5. Fine-grained Pattern Matching
  6. Dynamic Synchronized Multi-Modal EMA
  7. Experiments
  8. Datasets and Implementation
  9. Cross-Modal Retrieval Results
  10. Ablation Study
  11. Conclusion

Figures (3)

  • Figure 1: Overview of NeuroAlign framework architecture. The illustration demonstrates our approach using video, fMRI data pairs, highlighting the Neural-Temporal Contrastive Learning (NTCL) module and Vector Quantization with DynaSyncMM-EMA update mechanism. While shown with two modalities for clarity, our framework naturally extends to tri-modal integration, enabling comprehensive alignment across $\{$fMRI, Video, Caption$\}$
  • Figure 2: Visualization results of retrieval task. The highlighted areas represent the retrieved video embedding with high activation correspondence with the query fMRI embeddings.
  • Figure 3: t-SNE visualization comparing NeuroAlign (Ours) with CLIP-based baseline. Colors represent different modalities: fMRI (blue), video (orange), and text (green). NeuroAlign achieves significantly tighter cross-modal clustering compared to CLIP-based methods.