Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A multimodal LLM for the non-invasive decoding of spoken text from brain recordings

Youssef Hmamouche, Ismail Chihab, Lahoucine Kdouri, Amal El Fallah Seghrouchni

TL;DR

This work proposes and end-to-end multimodal LLM for decoding spoken text from fMRI signals that outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth.

Abstract

Brain-related research topics in artificial intelligence have recently gained popularity, particularly due to the expansion of what multimodal architectures can do from computer vision to natural language processing. Our main goal in this work is to explore the possibilities and limitations of these architectures in spoken text decoding from non-invasive fMRI recordings. Contrary to vision and textual data, fMRI data represent a complex modality due to the variety of brain scanners, which implies (i) the variety of the recorded signal formats, (ii) the low resolution and noise of the raw signals, and (iii) the scarcity of pretrained models that can be leveraged as foundation models for generative learning. These points make the problem of the non-invasive decoding of text from fMRI recordings very challenging. In this paper, we propose and end-to-end multimodal LLM for decoding spoken text from fMRI signals. The proposed architecture is founded on (i) an encoder derived from a specific transformer incorporating an augmented embedding layer for the encoder and a better-adjusted attention mechanism than that present in the state of the art, and (ii) a frozen large language model adapted to align the embedding of the input text and the encoded embedding of brain activity to decode the output text. A benchmark in performed on a corpus consisting of a set of interactions human-human and human-robot interactions where fMRI and conversational signals are recorded synchronously. The obtained results are very promising, as our proposal outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth. The implementation code is provided in https://github.com/Hmamouche/brain_decode.

A multimodal LLM for the non-invasive decoding of spoken text from brain recordings

TL;DR

This work proposes and end-to-end multimodal LLM for decoding spoken text from fMRI signals that outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth.

Abstract

Brain-related research topics in artificial intelligence have recently gained popularity, particularly due to the expansion of what multimodal architectures can do from computer vision to natural language processing. Our main goal in this work is to explore the possibilities and limitations of these architectures in spoken text decoding from non-invasive fMRI recordings. Contrary to vision and textual data, fMRI data represent a complex modality due to the variety of brain scanners, which implies (i) the variety of the recorded signal formats, (ii) the low resolution and noise of the raw signals, and (iii) the scarcity of pretrained models that can be leveraged as foundation models for generative learning. These points make the problem of the non-invasive decoding of text from fMRI recordings very challenging. In this paper, we propose and end-to-end multimodal LLM for decoding spoken text from fMRI signals. The proposed architecture is founded on (i) an encoder derived from a specific transformer incorporating an augmented embedding layer for the encoder and a better-adjusted attention mechanism than that present in the state of the art, and (ii) a frozen large language model adapted to align the embedding of the input text and the encoded embedding of brain activity to decode the output text. A benchmark in performed on a corpus consisting of a set of interactions human-human and human-robot interactions where fMRI and conversational signals are recorded synchronously. The obtained results are very promising, as our proposal outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth. The implementation code is provided in https://github.com/Hmamouche/brain_decode.
Paper Structure (34 sections, 6 equations, 4 figures, 3 tables)

This paper contains 34 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Decoding Images from Brain Recordings
  4. Decoding Text from Brain Recordings
  5. Method
  6. A Multimodal LLM
  7. A bibartite Transformer with inception deconvolution-based fMRI encoder
  8. Inception Deconvolution Embedding
  9. Simplex Attention
  10. Duplex Attention
  11. A Bipartite Transformer
  12. Experiments and Results
  13. Experimental Setup
  14. Dataset Description
  15. Data Availability
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Illustrative diagram of the interaction protocoal in the used dataset rauchbauer2019brain and the decoding process of our approach. The datasets consist of conversational and neurophysiological signals recorded during human-human and human-robot interactions. The recorded signals are then leveraged to construct generative models to decode the participant's text from his whole-brain fMRI recordings.
  • Figure 2: Schema of the proposed system - fMRI brain activity to text representation learning via two training stages: (1) brain activity and text mapping with an improved deconvolution bipartite transformer, and (2) multimodal generative pre-training using the trained encoder and a frozen large language model as a decoder.
  • Figure 3: Architecture of the proposed Bipartite Transformer for text decoding from fMRI brain activity.
  • Figure 4: Schema of the proposed MLLM with CLIP encoder for stimuli image integration. .