BrainCodec: Neural fMRI codec for the decoding of cognitive brain states

TL;DR

This study proposes BrainCodec, a novel fMRI codec inspired by the neural audio codec, and demonstrates that fMRI reconstructions using BrainCodec can enhance the visibility of brain activity by achieving higher SNR, suggesting its potential as a novel denoising method.

Abstract

Recently, leveraging big data in deep learning has led to significant performance improvements, as confirmed in applications like mental state decoding using fMRI data. However, fMRI datasets remain relatively small in scale, and the inherent issue of low signal-to-noise ratios (SNR) in fMRI data further exacerbates these challenges. To address this, we apply compression techniques as a preprocessing step for fMRI data. We propose BrainCodec, a novel fMRI codec inspired by the neural audio codec. We evaluated BrainCodec's compression capability in mental state decoding, demonstrating further improvements over previous methods. Furthermore, we analyzed the latent representations obtained through BrainCodec, elucidating the similarities and differences between task and resting state fMRI, highlighting the interpretability of BrainCodec. Additionally, we demonstrated that fMRI reconstructions using BrainCodec can enhance the visibility of brain activity by achieving higher SNR, suggesting its potential as a novel denoising method. Our study shows that BrainCodec not only enhances performance over previous methods but also offers new analytical possibilities for neuroscience. Our codes, dataset, and model weights are available at https://github.com/amano-k-lab/BrainCodec.

Figures (17)

• Figure 1: The architecture of BrainCodec: This model consists of an Encoder, a Decoder, and a Residual Vector Quantization (RVQ) module. The model input is data $X \in \mathbb{R}^{t\times 1024}$, transformed by DiFuMo from raw BOLD signals. It is trained through two losses: reconstruction error $\ell_r$ and commitment loss $\ell_w$ in RVQ.
• Figure 2: Upstream learning of CSM with BrainCodec. BrainCodec is fixed. The training objective is L1 reconstruction loss ($\ell_r$).
• Figure 3: Downstream learning of CSM with BrainCodec. BrainCodec is also fixed. The training objective is cross-entropy loss ($\ell_c$).
• Figure 4: The UMAP visualization compares Task-codebook (red), Rest-codebook (blue), and Random-codebook (green). The left plot shows the first codebook, while the right plot shows the second codebook.
• Figure 5: Comparison of the original fMRI from a MOTOR task trial (tr=0.72) in the HCP dataset with its reconstructed fMRI using BrainVAE, BrainCodec, and the averaged fMRI over 78 runs (Mean). BrainCodec uses the full codebooks (0-7), and the first half (0-3). The regions encircled by the orange indicate the visual cortex, while the green denote the tongue motor area.