Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models

TL;DR

This work tackles data inefficiency and per-participant limitations in aligning pretrained speech models with human brain responses. It introduces Multi-brain-tuning, a scalable fine-tuning framework that jointly adapts Wav2Vec2.0 and HuBERT using fMRI data from multiple participants via a shared projection head and LoRA updates. Results show a five-fold reduction in required fMRI data for new participants, up to a 50% gain in overall brain alignment, and strong generalization to a new dataset, with concomitant improvements in downstream semantic tasks. The approach demonstrates a bidirectional benefit between neuroscience and AI, enabling population-level brain alignment and more generalizable semantic representations, with code and models released for reproducibility.

Abstract

Pretrained language models are remarkably effective in aligning with human brain responses elicited by natural language stimuli, positioning them as promising model organisms for studying language processing in the brain. However, existing approaches for both estimating and improving this brain alignment are participant-dependent and highly affected by the amount of data available per participant, hindering both generalization to new participants and population-level analyses. In this work, we address these limitations by introducing a scalable, generalizable brain-tuning method, in which we fine-tune pretrained speech language models to jointly predict fMRI responses from multiple participants. We demonstrate that the resulting brain-tuned models exhibit strong individual brain alignment while generalizing across participants. Specifically, our method leads to 1) a 5-fold decrease in the amount of fMRI data needed to predict brain data from new participants, 2) up to a 50% increase in the overall brain alignment, and 3) strong generalization to new unseen datasets. Furthermore, this multi-participant brain-tuning additionally improves downstream performance on semantic tasks, suggesting that training using brain data from multiple participants leads to more generalizable semantic representations. Taken together, these findings demonstrate a bidirectional benefit between neuroscience and AI, helping bridge the gap between the two fields. We make our code and models publicly available at https://github.com/bridge-ai-neuro/multi-brain-tuning.

Figures (12)

• Figure 1: Multi-brain-tuning Approach. Given participant responses, we project them to a common space (see Sec.\ref{['method:train_data']}). The aligned responses of multiple participants are then used to fine-tune the pretrained model with low-rank updates, binding by the shared stimulus as detailed in Sec.\ref{['method:miltip_method']}.
• Figure 2: Brain encoding efficiency for both training and held-out participants and two model families (Wav2Vec2.0 and HuBERT). Brain-tuned models consistently outperform pretrained counterparts on both training and held-out participants. Multi-brain-tuned models consistently outperform Single-brain-tuned ones, reaching the max. pretrained performance with approximately a fifth of the brain encoding data and improving brain alignment up to $50\%$ from the max. pretrained brain alignment.
• Figure 3: Brain-tuning generalization. (A) Scaling with tuning data size, showing that Multi-brain-tuning scales better, especially on held-out participants. (B) Voxel-wise change in brain alignment after Multi-brain-tuning for a held-out participant from the Moth Radio hour dataset (the remaining participants are reported in Supp.\ref{['app:res_brain']}), showing improved alignment across frontal and parietal regions. (C) Alignment improvement on novel stimuli and participants (from the Narratives dataset), indicating the capability of Multi-brain-tuned models for cross-dataset generalization.
• Figure 4: Scaling of downstream performance with tuning data size. Brain-tuned models' performance increases with more data, with the Multi-brain-tuned taking less data to match the LLM-tuned model.
• Figure 5: Effect of varying the loss function and LoRA rank on Multi-brain-tuning with (A) showing that Rank-8 updates perform best, and (B) indicating that the $L_2$ loss scales better.