Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information

TL;DR

The paper tackles how visual memory is represented during continuous viewing by decoding past semantic content from fMRI. It introduces Memory Disentangling, a task to extract current and past visual information while separating interfering memory components, and proposes a straightforward multi-MLP baseline plus a disentangled contrastive learning framework inspired by proactive interference. Using the NSD dataset, the authors show that brain signals retain memory for roughly $3$–$4$ items, and that the disentangled loss improves current-time semantic decoding though past-memory extraction remains challenging. The approach has potential implications for brain-computer interfaces and addressing fMRI’s low temporal resolution, while also highlighting limitations in past-memory interpretability and generalization to other memory modalities.

Abstract

The human visual system is capable of processing continuous streams of visual information, but how the brain encodes and retrieves recent visual memories during continuous visual processing remains unexplored. This study investigates the capacity of working memory to retain past information under continuous visual stimuli. And then we propose a new task Memory Disentangling, which aims to extract and decode past information from fMRI signals. To address the issue of interference from past memory information, we design a disentangled contrastive learning method inspired by the phenomenon of proactive interference. This method separates the information between adjacent fMRI signals into current and past components and decodes them into image descriptions. Experimental results demonstrate that this method effectively disentangles the information within fMRI signals. This research could advance brain-computer interfaces and mitigate the problem of low temporal resolution in fMRI.

Figures (13)

• Figure 1: The schematic diagram of Memory Disentangling based on decoding image semantic information.
• Figure 2: Overview of visual memory analysis. (a) Acquisition of continuous visual stimuli data, including image embeddings and fMRI signals. (b) Ridge regression analysis for visual memory, where"R" represents the ridge regression model, and $k$ is offset. The figure illustrates an example for $k=2$. (c) Trail-wise RSA, with the meaning of k remaining consistent with the previous context. Note that, for explanatory purposes, the size of the RDMs in the figure is illustrative and not representative of the actual size.
• Figure 3: The results of ridge regression analysis for four participants.
• Figure 4: The trail-wise RSA results for four participants.
• Figure 5: A schematic diagram of the straightforward approach using separate MLPs.