Attention-Based Variational Framework for Joint and Individual Components Learning with Applications in Brain Network Analysis

TL;DR

CM-JIVNet tackles the challenge of integrating structural and functional brain connectivity by learning a nonlinear, probabilistic latent space that factorizes into joint and modality-specific components. It uses dual VAEs with an attention-based fusion module and an orthogonality- constrained joint–individual separation to enable joint generation and interpretable disentanglement of SC and FC. The approach achieves state-of-the-art cross-modal reconstruction and missing-modality prediction on 1,065 HCP-YA subjects, and its supervised variant (sCM-JIVNet) substantially improves behavioral trait prediction by ~0.11 in correlation, with strongest gains for fluid intelligence and language-related tasks. This framework advances scalable, interpretable multimodal brain analysis and lays groundwork for biomarker discovery across diverse cohorts and modalities.

Abstract

Brain organization is increasingly characterized through multiple imaging modalities, most notably structural connectivity (SC) and functional connectivity (FC). Integrating these inherently distinct yet complementary data sources is essential for uncovering the cross-modal patterns that drive behavioral phenotypes. However, effective integration is hindered by the high dimensionality and non-linearity of connectome data, complex non-linear SC-FC coupling, and the challenge of disentangling shared information from modality-specific variations. To address these issues, we propose the Cross-Modal Joint-Individual Variational Network (CM-JIVNet), a unified probabilistic framework designed to learn factorized latent representations from paired SC-FC datasets. Our model utilizes a multi-head attention fusion module to capture non-linear cross-modal dependencies while isolating independent, modality-specific signals. Validated on Human Connectome Project Young Adult (HCP-YA) data, CM-JIVNet demonstrates superior performance in cross-modal reconstruction and behavioral trait prediction. By effectively disentangling joint and individual feature spaces, CM-JIVNet provides a robust, interpretable, and scalable solution for large-scale multimodal brain analysis.

Figures (6)

• Figure 1: Overview of the CM-JIVNet workflow. CM-JIVNet comprises four main modules. (1) Dual variational encoders extract latent Gaussian parameters from the FC and SC connectivity matrices. (2) Attention-based fusion aligns the modality-specific latent spaces through a multi-head self-attention and gating mechanism, capturing nonlinear SC–FC dependencies. (3) Joint–individual separation decomposes the fused latent representation into three orthogonal components—joint, FC-specific, and SC-specific—enhancing interpretability of cross-modal coupling. (4) Dual decoders and supervision (optional) reconstruct each modality from its corresponding latent factors while separate branch (optional) encourages prediction of traits from the learned latent factors. During training, reconstruction and prediction losses are optimized jointly under variational and orthogonality regularization constraints.
• Figure 2: t-SNE embedding of concatenated latent representations, colored by subspace.
• Figure 3: Edge-level SC-FC coupling induced by a PC$_1$ traversal in the joint latent space. Each point corresponds to one undirected edge. The horizontal and vertical axes report endpoint changes $\Delta\mathrm{SC}_{ij}$ and $\Delta\mathrm{FC}_{ij}$, respectively, between the traversal extremes (e.g., $\pm 3\sigma$ along PC$_1$). Quadrants categorize coupling signatures: $\mathrm{FC}\uparrow/\mathrm{SC}\uparrow$ (concordant increase), $\mathrm{FC}\uparrow/\mathrm{SC}\downarrow$ (decoupling), and the remaining sign combinations. A dominant mass in the $\mathrm{FC}\uparrow/\mathrm{SC}\uparrow$ quadrant indicates that the leading joint axis encodes a coupled mode of variation, while the minority decoupling edges highlight edge-specific departures from monotone SC--FC co-variation.
• Figure 4: Spatial distribution of per-edge change rates along the joint latent PC$_1$. Each matrix entry is the least-squares slope with respect to the PC$_1$ coordinate, $b^{\mathrm{FC}}_{ij}$ for FC and $b^{\mathrm{SC}}_{ij}$ for $\log(1+\mathrm{SC})$.Positive values (red) indicate edges that strengthen as PC$_1$ increases, while negative values (blue) indicate weakening
• Figure 5: Comparison of behavioral trait prediction performances between unsupervised and supervised CM-JIVNet models using concatenated latent representations.