Beyond Pixels: A Differentiable Pipeline for Probing Neuronal Selectivity in 3D

TL;DR

Current approaches to neural selectivity infer from 2D pixels, making it difficult to isolate 3D geometry and lighting effects. The authors introduce a differentiable rendering pipeline that optimizes 3D meshes and scene parameters to maximize neuronal predictions $f(I)$, producing 3D-Maximally Exciting Inputs (3D-MEIs) rendered by $R(M)$. They implement mesh deformations via $K$ radial basis function kernels with learnable scales $\sigma_k$ and offsets $\delta_k$ and constrain deformations with Laplacian, edge-length, area, and ARAP regularizers, optimizing with Adam. Experiments on macaque V4 encoding models show that 3D-MEIs resemble pixel MEIs but enable explicit manipulation of pose and lighting, revealing geometry-tuned invariances and suggesting broader potential for probing physically grounded, 3D stimuli in vision neuroscience.

Abstract

Visual perception relies on inference of 3D scene properties such as shape, pose, and lighting. To understand how visual sensory neurons enable robust perception, it is crucial to characterize their selectivity to such physically interpretable factors. However, current approaches mainly operate on 2D pixels, making it difficult to isolate selectivity for physical scene properties. To address this limitation, we introduce a differentiable rendering pipeline that optimizes deformable meshes to obtain MEIs directly in 3D. The method parameterizes mesh deformations with radial basis functions and learns offsets and scales that maximize neuronal responses while enforcing geometric regularity. Applied to models of monkey area V4, our approach enables probing neuronal selectivity to interpretable 3D factors such as pose and lighting. This approach bridges inverse graphics with systems neuroscience, offering a way to probe neural selectivity with physically grounded, 3D stimuli beyond conventional pixel-based methods.

Figures (3)

• Figure 1: A. Schematic overview of the differentiable rendering pipeline. Starting from an initial mesh and scene parameters (lighting, camera, background), the pipeline iteratively deforms the mesh using radial basis function offsets and optimizes scene parameters to maximize model responses. Losses and regularizers enforce smoothness, uniform surface areas, edge-length preservation, and locally rigid deformations. B. 3D Maximally exciting images (3D-MEIs) generated for multiple selected model macaque V4 neurons. For each selected neuron (unit), we show the pixel based EGG MEI and the 3D-MEI, as well as their 3D views. Right: Bar plots comparing activation values for EGG and 3D-MEI. A 2D-to-3D model (VGGT) applied to EGG MEIs fails to generate meaningful 3D structures.
• Figure 2: Response profile of Unit 521 to variations in 3D pose and lighting direction. A. Heatmap of model responses to pose variations (azimuth × elevation) relative to the original view, showing tolerance to moderate rotations. B. Light-direction tuning of the same unit. Responses are shown for selected light positions on the hemisphere, grouped into high, mid, and low response sets (labels indicate azimuth/elevation in degrees and response values).
• Figure 3: Validation of the pipeline on simulated data. A. RBF-based mesh deformation produces complex targets such as the Stanford Bunny (from a sphere) and Utah Teapot (from a torus) by minimizing Chamfer loss ravi2020pytorch3d between the input and target meshes. B. Optimization curves for simulated simple and complex cells with “sheet” and “sphere” initializations approach to the upper bound of 1. C. 3D-MEI results for a simulated simple cell a. and complex cell b.. Outputs recover the ground-truth filters; for the phase-invariant complex cell, different initializations yield Gabor-like patterns in distinct phases (example shown). D. World setup of the differentiable renderer R, showing world and camera axes. The base ring (z=0) marks the base of half-dome used in light-direction tuning.