BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM

TL;DR

BP-GPT tackles the problem of decoding auditory language from fMRI by introducing an end-to-end framework that uses fMRI-derived prompts to steer GPT-2. The method features a two-stage training regime with a text-prompt to bridge modality differences, and a contrastive learning objective to align fMRI prompts with the optimal text prompt. Empirical results on an open auditory semantic decoding dataset show consistent improvements in METEOR and BERTScore over the prior state-of-the-art, demonstrating the feasibility and advantage of prompting LLMs with brain signals for auditory decoding. The approach is adaptable to future LLM advances and comes with publicly available code for reproducibility and extension.

Abstract

Decoding language information from brain signals represents a vital research area within brain-computer interfaces, particularly in the context of deciphering the semantic information from the fMRI signal. Although existing work uses LLM to achieve this goal, their method does not use an end-to-end approach and avoids the LLM in the mapping of fMRI-to-text, leaving space for the exploration of the LLM in auditory decoding. In this paper, we introduce a novel method, the Brain Prompt GPT (BP-GPT). By using the brain representation that is extracted from the fMRI as a prompt, our method can utilize GPT-2 to decode fMRI signals into stimulus text. Further, we introduce the text prompt and align the fMRI prompt to it. By introducing the text prompt, our BP-GPT can extract a more robust brain prompt and promote the decoding of pre-trained LLM. We evaluate our BP-GPT on the open-source auditory semantic decoding dataset and achieve a significant improvement up to 4.61 on METEOR and 2.43 on BERTScore across all the subjects compared to the state-of-the-art method. The experimental results demonstrate that using brain representation as a prompt to further drive LLM for auditory neural decoding is feasible and effective. The code is available at https://github.com/1994cxy/BP-GPT.

Figures (2)

• Figure 1: Two training stages of our method.
• Figure 2: Ground Truth (top) and the decoding result (bottom) from the test set. $ is added according to the fMRI TR as the alignment token.