fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment

TL;DR

This work addresses the challenge of decoding co-speech gestures from fMRI, a setting with scarce paired brain-gesture data. It introduces fMRI2GES, a two-phase framework that first trains F2T and T2G with supervised data and then refines F2G through Dual Brain Decoding Alignment using pseudo-labels from cascaded models, all within a diffusion-model-based gesture generator and cross-attention architecture. The study demonstrates gesture reconstruction from fMRI signals, investigates ROI contributions (notably auditory and speech regions), and shows favorable human judgments compared to baselines, indicating promising implications for BCIs and multi-modal brain decoding. Overall, it provides a principled approach to bridge brain signals, language, and gestures, advancing neuroscience and cognitive science while offering practical insights for HCI systems.

Abstract

Understanding how the brain responds to external stimuli and decoding this process has been a significant challenge in neuroscience. While previous studies typically concentrated on brain-to-image and brain-to-language reconstruction, our work strives to reconstruct gestures associated with speech stimuli perceived by brain. Unfortunately, the lack of paired \{brain, speech, gesture\} data hinders the deployment of deep learning models for this purpose. In this paper, we introduce a novel approach, \textbf{fMRI2GES}, that allows training of fMRI-to-gesture reconstruction networks on unpaired data using \textbf{Dual Brain Decoding Alignment}. This method relies on two key components: (i) observed texts that elicit brain responses, and (ii) textual descriptions associated with the gestures. Then, instead of training models in a completely supervised manner to find a mapping relationship among the three modalities, we harness an fMRI-to-text model, a text-to-gesture model with paired data and an fMRI-to-gesture model with unpaired data, establishing dual fMRI-to-gesture reconstruction patterns. Afterward, we explicitly align two outputs and train our model in a self-supervision way. We show that our proposed method can reconstruct expressive gestures directly from fMRI recordings. We also investigate fMRI signals from different ROIs in the cortex and how they affect generation results. Overall, we provide new insights into decoding co-speech gestures, thereby advancing our understanding of neuroscience and cognitive science.

Figures (4)

• Figure 1: Our work aims to directly reconstruct gestures from brain signals, overcoming the challenge of lacking paired brain, text, and gesture data. We achieve this by leveraging texts that elicit brain activity and corresponding textual descriptions of the gestures.
• Figure 2: An overview of our fMRI2GES framework. In the first phase, we conduct supervised training that focuses on learning an fMRI-to-text (F2T) model and a text-to-gesture (T2G) model. In the second phase, we leverage the Dual Brain Decoding Alignment mechanism to train the fMRI-to-gesture (F2G) model in an unsupervised learning manner.
• Figure 3: Gesture examples were generated using T2G and F2G. F2G (All) utilizes voxel-sampled fMRI data from the entire brain, F2G (Auditory) focuses on auditory regions, and F2G (Speech) focuses on speech-related areas. Additionally, F2G (S+A) integrates fMRI data from both language and auditory regions.
• Figure 4: Brain activity prediction performance assessed using Pearson's correlation coefficients, mapped onto the flattened cortical surface with the occipital areas at the center, covering both the left and right hemispheres. The prediction was conducted based on latent gesture representations extracted from the Unet in F2G, conditioned on fMRI data from the auditory cortex (upper) and speech-related cortices (lower). Brain areas: ① EBA (extrastriate body area), ② AC (auditory cortex), ③ IPS (intraparietal sulcus), ④ Broca (Broca's area), ⑤ IFSFP (inferior frontal sulcus face patch), ⑥ S1H (primary somatosensory cortex for hands), ⑦ M1H (primary motor cortex for hands), ⑧ FEF (frontal eye fields).