Toward High-Fidelity Visual Reconstruction: From EEG-Based Conditioned Generation to Joint-Modal Guided Rebuilding

Abstract

Human visual reconstruction aims to reconstruct fine-grained visual stimuli based on subject-provided descriptions and corresponding neural signals. As a widely adopted modality, Electroencephalography (EEG) captures rich visual cognition information, encompassing complex spatial relationships and chromatic details within scenes. However, current approaches are deeply coupled with an alignment framework that forces EEG features to align with text or image semantic representation. The dependency may condense the rich spatial and chromatic details in EEG that achieved mere conditioned image generation rather than high-fidelity visual reconstruction. To address this limitation, we propose a novel Joint-Modal Visual Reconstruction (JMVR) framework. It treats EEG and text as independent modalities for joint learning to preserve EEG-specific information for reconstruction. It further employs a multi-scale EEG encoding strategy to capture both fine- and coarse-grained features, alongside image augmentation to enhance the recovery of perceptual details. Extensive experiments on the THINGS-EEG dataset demonstrate that JMVR achieves SOTA performance against six baseline methods, specifically exhibiting superior capabilities in modeling spatial structure and chromatic fidelity.

Figures (8)

• Figure 1: Overall Framework. The EEG signals and corresponding textual descriptions are input to the joint latent space in JMVR for training and visual reconstruction.
• Figure 2: Overview of the JMVR framework. Text prompts are processed into coarse- and fine-grained embeddings. The former combines with diffusion timesteps and EEG to construct the condition latent for feature modulation, while the latter is concatenated with EEG representations to generate the cognition latent. Within the JMVR Block, this cognition latent interacts with the enhanced image latent via a joint-attention mechanism to synthesize the final output after modulation.
• Figure 3: Image Augmentation Process. The edge detection map, saturation heatmap, and depth map were extracted in advance and then combined with the original image to form the augmented image set for each sample.
• Figure 4: Visual comparison of reconstruction quality (The reconstruction samples were selected from subject-08).
• Figure 5: 3D point clouds of the reconstruction image (The sample was selected from the aircraft carrier reconstruction by Subject-08).