Image-to-Brain Signal Generation for Visual Prosthesis with CLIP Guided Multimodal Diffusion Models

TL;DR

The paper tackles the challenge of image-to-brain signal generation to advance brain encoding in visual prostheses. It introduces a diffusion-transformer framework (DiT) with cross-attention that aligns brain-signal embeddings to unified CLIP-based visual-semantic embeddings, enriched by LLM-generated captions. A learnable spatio-temporal position encoding models both brain regions and temporal dynamics, and DDIM enables efficient sampling. Experiments on THINGS-EEG2 and THINGS-MEG show improved quantitative metrics and biologically plausible signals, with ablations confirming the importance of CLIP-text guidance and occipital-region contributions, while cross-subject generalization remains a challenge with real-world implications for deployment.

Abstract

Visual prostheses hold great promise for restoring vision in blind individuals. While researchers have successfully utilized M/EEG signals to evoke visual perceptions during the brain decoding stage of visual prostheses, the complementary process of converting images into M/EEG signals in the brain encoding stage remains largely unexplored, hindering the formation of a complete functional pipeline. In this work, we present a novel image-to-brain signal framework that generates M/EEG from images by leveraging the diffusion transformer architecture enhanced with cross-attention mechanisms. Specifically, we employ a diffusion transformer (DiT) architecture based on denoising diffusion implicit models (DDIM) to achieve brain signal generation. To realize the goal of image-to-brain signal conversion, we use cross-attention mechanisms to align brain signal embeddings with CLIP image embeddings. Moreover, we leverage large language models (LLMs) to generate image captions, and concatenate the resulting CLIP text embeddings with CLIP image embeddings to form unified embeddings for cross-attention alignment, enabling our model to capture core semantic information. Moreover, to capture core semantic information, we use large language models (LLMs) to generate descriptive and semantically accurate captions for images. Furthermore, we introduce a learnable spatio-temporal position encoding that combines brain region embeddings with temporal embeddings to capture both spatial and temporal characteristics of brain signals. We evaluate the framework on two multimodal benchmark datasets (THINGS-EEG2 and THINGS-MEG) and demonstrate that it generates biologically plausible brain signals.

Figures (8)

• Figure 1: The framework of the visual prostheses. Visual prostheses utilize an image sensor to capture natural scenes. A processing framework takes the recorded signals as input and predicts the stimuli for the retinal prosthesis. A phosphene model receives stimulation from the implanted prosthesis and evokes visual perception (or 'phosphene'). The performance of the framework is evaluated by comparing the similarity between the original image and the visual perception.
• Figure 2: Overall architecture of our image-to-brain framework. The framework consists of: (1) a CLIP visual encoder (ViT-L/14) that extracts CLIP image embeddings, (2) a large language model (Qwen2-VL-2B-Instruct) that generates image captions which are then encoded by CLIP text encoder to obtain CLIP text embeddings, (3) the concatenation of CLIP image and text embeddings along the token dimension to form unified visual-semantic embeddings, and (4) a Diffusion Transformer (DiT) based on DDIM that generates brain signals through cross-attention mechanisms, where brain signal patch embeddings serve as Query and unified embeddings serve as Key and Value. The model incorporates learnable spatio-temporal position embeddings that combine brain region embeddings and temporal embeddings to capture the spatio-temporal characteristics of brain signals (detailed encoding methods are provided in Appendix \ref{['position_embeddings']}).
• Figure 3: Cross-subject topography comparison for THINGS-EEG2. The figure shows EEG topographies comparing Subject 1 (training) and Subject 2 (test). The third row shows the difference between training and test subjects, highlighting inter-individual variability in brain signals.
• Figure 4: An example of an image caption generated by Qwen2-VL-2B-Instruct.
• Figure 5: CLIP Score distributions of LLM-generated captions.