Uncovering Semantic Selectivity of Latent Groups in Higher Visual Cortex with Mutual Information-Guided Diffusion

TL;DR

The paper tackles how object-centered visual information is represented in higher visual cortex by introducing MIG-Vis, which combines a group-wise disentangled VAE to extract neural latent groups with a mutual-information-guided diffusion framework to visualize their semantic attributes. It demonstrates that latent groups in IT cortex encode distinct semantic factors, such as intra-category pose, inter-category semantics, and intra-category content details, and validates these findings through deterministic DDIM-based editing guided by MI. The approach yields high neural reconstruction quality and significantly higher latent disentanglement than standard VAEs, with robust quantitative and qualitative evidence from macaque IT data. This work provides direct interpretable insight into the structured, multi-dimensional nature of visual coding in primate cortex and offers a reproducible tool for exploring neural subspace geometry.

Abstract

Understanding how neural populations in higher visual areas encode object-centered visual information remains a central challenge in computational neuroscience. Prior works have investigated representational alignment between artificial neural networks and the visual cortex. Nevertheless, these findings are indirect and offer limited insights to the structure of neural populations themselves. Similarly, decoding-based methods have quantified semantic features from neural populations but have not uncovered their underlying organizations. This leaves open a scientific question: "how feature-specific visual information is distributed across neural populations in higher visual areas, and whether it is organized into structured, semantically meaningful subspaces." To tackle this problem, we present MIG-Vis, a method that leverages the generative power of diffusion models to visualize and validate the visual-semantic attributes encoded in neural latent subspaces. Our method first uses a variational autoencoder to infer a group-wise disentangled neural latent subspace from neural populations. Subsequently, we propose a mutual information (MI)-guided diffusion synthesis procedure to visualize the specific visual-semantic features encoded by each latent group. We validate MIG-Vis on multi-session neural spiking datasets from the inferior temporal (IT) cortex of two macaques. The synthesized results demonstrate that our method identifies neural latent groups with clear semantic selectivity to diverse visual features, including object pose, inter-category transformations, and intra-class content. These findings provide direct, interpretable evidence of structured semantic representation in the higher visual cortex and advance our understanding of its encoding principles.

Figures (9)

• Figure 1: Single-neuron linear decoding results in IT cortex of two macaques during a passive object recognition task.(A) Decoding results on both two macaques reveal that IT neurons exhibit mixed selectivity. The connecting line thickness denotes the weight scale. These neurons contribute to both low-level pose attributes (e.g., rotation) and high-level semantic features (e.g., category id). (B) Images with variations along each visual-semantic feature.
• Figure 2: (A) The semantic image editing procedure. This consists of a deterministic forward process from $t=0$ to an intermediate timestep $t=t'$, followed by a neural-guided deterministic synthesis process back to $t=0$. (B) A schematic of the Mutual Information (MI) Landscape. This landscape is defined by our guidance objective, which maximizes the MI between a synthesized image and the neural latent group $\mathbf{z}_1$ of the target image (encoding object pose).
• Figure 3: Macaques IT cortex dataset. (A) Experimental Setting. (B) Example images from the eight object categories in the dataset.
• Figure 4: Synthesized images by MIG-Vis under varying guidance strengths. (A) Results using a frontal-view face and table as the original image, guiding toward a pear. (B) Results using a car and hedgehog object as the original image, guiding toward a plane.
• Figure 5: Comparison of images synthesized by MIG-Vis and baseline methods. (A) Results from probing Latent Group 1 (pose). (B) Results from probing Latent Group 2 (inter-category).