NeuroVolve: Evolving Visual Stimuli toward Programmable Neural Objectives

TL;DR

NeuroVolve presents a flexible framework for brain-guided image synthesis that optimizes in the embedding space of a vision-language model to satisfy programmable neural objectives across brain regions. By coupling a voxelwise encoding model with embedding-space optimization and a diffusion-based image generator, it yields semantic trajectories that unify image editing and preferred-stimulus generation while capturing subject-specific neural tuning. The approach recovers known region selectivity, reveals low-level feature tuning in early visual areas, and enables complex, multi-region constraints such as co-activation and suppression. This data-driven, personalized synthesis tool offers a new avenue for mapping visual representations and potential brain-computer interface applications.

Abstract

What visual information is encoded in individual brain regions, and how do distributed patterns combine to create their neural representations? Prior work has used generative models to replicate known category selectivity in isolated regions (e.g., faces in FFA), but these approaches offer limited insight into how regions interact during complex, naturalistic vision. We introduce NeuroVolve, a generative framework that provides brain-guided image synthesis via optimization of a neural objective function in the embedding space of a pretrained vision-language model. Images are generated under the guidance of a programmable neural objective, i.e., activating or deactivating single regions or multiple regions together. NeuroVolve is validated by recovering known selectivity for individual brain regions, while expanding to synthesize coherent scenes that satisfy complex, multi-region constraints. By tracking optimization steps, it reveals semantic trajectories through embedding space, unifying brain-guided image editing and preferred stimulus generation in a single process. We show that NeuroVolve can generate both low-level and semantic feature-specific stimuli for single ROIs, as well as stimuli aligned to curated neural objectives. These include co-activation and decorrelation between regions, exposing cooperative and antagonistic tuning relationships. Notably, the framework captures subject-specific preferences, supporting personalized brain-driven synthesis and offering interpretable constraints for mapping, analyzing, and probing neural representations of visual information.

Figures (31)

• Figure 1: Images generated using NeuroVolve. We visualize image trajectories optimized to activate specific brain regions. Starting from random seeds, NeuroVolve reveals region-selective features over iterations—recovering known high-level selectivity (e.g., FFA, PPA, EBA) and capturing low level sfeatures selectivity in early areas (e.g., V1, V2). It also supports compositional objectives, generating coherent images under complex multi-region constraints.
• Figure 2: Overview of the NeuroVolve framework. (a) Starting from an initial image, we extract Q-Former embeddings using a frozen BLIP-2 encoder. These embeddings are then iteratively optimized under a neural objective using gradients from a learned subject-specific MLP $\Phi$, which maps Q-Former embeddings to voxel-wise brain activity. The resulting embedding trajectory is used to guide a diffusion model (BLIP-Diffusion) for image synthesis. (b) Visualization of optimization in the Q-Former embedding space. NeuroVolve minimizes a neural objective loss to evolve the embedding toward a region-selective optimum (e.g., maximizing FFA). The resulting semantic trajectory produces interpretable, brain-aligned images.
• Figure 3: Region-specific image evolution under neural objectives (S5). Starting from the same seed images (left column), we optimize toward a neural objective of maximizing voxel activity in six distinct visual ROIs: FFA (faces), EBA (bodies), PPA (places), mfswords (words), V1 (edges, contrast), and V4 (color, curvature). NeuroVolve successfully evolves each input into semantically distinct outputs, reflecting the known selectivity of each region. Notably, our method captures both high-level semantic preferences (e.g., faces for FFA) and low-level visual features (e.g., high spatial frequency for V1, rich colors and curvature for V4). Demonstrating fine-grained, region-specific tuning across the visual hierarchy.
• Figure 4: NeuroVolve optimization trajectory for FFA. The plot shows progress toward maximizing FFA activation. Images sampled along the trajectory reveal the semantic evolution from the initial input to a face-like stimulus. Illustrating how optimization reveals interpretable visual transitions under neural objective guidance.
• Figure 5: Evaluating the distribution of NeuroVolve final images using an alternative encoder (ViT-H/14 CLIP). We retrain the voxel-wise encoding model using a different vision backbone (ViT-H/14 CLIP) and evaluate predicted responses across image sets. For each region, we compare three distributions: (1) all NSD images seen by the subject, (2) the top-100 NSD images ranked by the ViT-H/14-based encoder, and (3) the top-100 NeuroVolve-generated images ranked by the same encoder. NeuroVolve images consistently achieve higher predicted activations. Pushing responses beyond those evoked by natural stimuli, demonstrating the model’s capacity to synthesize maximally activating content.