NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Text
Yanting Yang, Xiaoxiao Li
TL;DR
This work tackles the problem of cross-modal alignment between fMRI functional connectivity and phenotypic text by introducing NeuroLIP, a framework that jointly learns region-level, token-conditioned representations. It leverages Text Token-Conditioned Attention (TTCA) to bind brain-region activity to individual text tokens and Cross-Modal Alignment via Localized Tokens (CALT) to align localized image-text pairs, while incorporating a sensitive-attribute disentanglement loss and a negative-gradient technique to curb demographic biases. The model is evaluated on ABIDE and ADHD-200, achieving superior fairness metrics without sacrificing standard predictive performance, and produces attention maps that align with known neuroanatomical patterns and meta-analytic results. Collectively, NeuroLIP advances transparent and equitable neuroimaging AI by enabling interpretable, region-specific cross-modal associations and robust bias mitigation, with potential impact on clinical neuropsychiatric research and diagnosis.
Abstract
Integrating functional magnetic resonance imaging (fMRI) connectivity data with phenotypic textual descriptors (e.g., disease label, demographic data) holds significant potential to advance our understanding of neurological conditions. However, existing cross-modal alignment methods often lack interpretability and risk introducing biases by encoding sensitive attributes together with diagnostic-related features. In this work, we propose NeuroLIP, a novel cross-modal contrastive learning framework. We introduce text token-conditioned attention (TTCA) and cross-modal alignment via localized tokens (CALT) to the brain region-level embeddings with each disease-related phenotypic token. It improves interpretability via token-level attention maps, revealing brain region-disease associations. To mitigate bias, we propose a loss for sensitive attribute disentanglement that maximizes the attention distance between disease tokens and sensitive attribute tokens, reducing unintended correlations in downstream predictions. Additionally, we incorporate a negative gradient technique that reverses the sign of CALT loss on sensitive attributes, further discouraging the alignment of these features. Experiments on neuroimaging datasets (ABIDE and ADHD-200) demonstrate NeuroLIP's superiority in terms of fairness metrics while maintaining the overall best standard metric performance. Qualitative visualization of attention maps highlights neuroanatomical patterns aligned with diagnostic characteristics, validated by the neuroscientific literature. Our work advances the development of transparent and equitable neuroimaging AI.