FOVI: A biologically-inspired foveated interface for deep vision models

TL;DR

FOVI proposes a biologically-inspired foveated interface that maps retina-like nonuniform sampling to a uniform sensor manifold, enabling kNN-convolution and efficient processing for deep vision models. It introduces FOVI-CNNs and FOVI-ViTs that learn over the foveated representation, using kernel mapping and LoRA adaptation to achieve competitive accuracy with a fraction of the computation at high resolutions. The framework reproduces key primate receptive-field properties, offers tunable foveation through a single parameter, and demonstrates practical benefits for efficient egocentric and active sensing, with open-source code and pretrained models. Overall, FOVI provides a general, end-to-end approach to scalable, high-resolution vision by balancing central acuity with peripheral context.

Abstract

Human vision is foveated, with variable resolution peaking at the center of a large field of view; this reflects an efficient trade-off for active sensing, allowing eye-movements to bring different parts of the world into focus with other parts of the world in context. In contrast, most computer vision systems encode the visual world at a uniform resolution, raising challenges for processing full-field high-resolution images efficiently. We propose a foveated vision interface (FOVI) based on the human retina and primary visual cortex, that reformats a variable-resolution retina-like sensor array into a uniformly dense, V1-like sensor manifold. Receptive fields are defined as k-nearest-neighborhoods (kNNs) on the sensor manifold, enabling kNN-convolution via a novel kernel mapping technique. We demonstrate two use cases: (1) an end-to-end kNN-convolutional architecture, and (2) a foveated adaptation of the foundational DINOv3 ViT model, leveraging low-rank adaptation (LoRA). These models provide competitive performance at a fraction of the computational cost of non-foveated baselines, opening pathways for efficient and scalable active sensing for high-resolution egocentric vision. Code and pre-trained models are available at https://github.com/nblauch/fovi and https://huggingface.co/fovi-pytorch.

Figures (18)

• Figure 1: A. Foveated sensing as uniform sampling on a magnified sensor manifold rovamo_isotropy_1984. With manifold divided at the vertical meridian and flattened, its relationship to the two hemispheres of primary visual cortex is evident. B. Building hierarchical convolutional networks via uniform k-nearest-neighbor (kNN) sampling on the sensor manifold. Kernel mapping allows for kNN-convolution over the sensor manifold, yielding denser and smaller RFs in the fovea. C. Building vision transformers (ViTs) from a kNN-convolution-based patchification of the sensor manifold. Low-rank adaptation allows for successful adaptation of off-the-shelf foundation models as efficient foveated variants.
• Figure 2: The relationship between cortical magnification and isotropic foveated sensing. A. The cortical magnification function commonly used to account for the organization of retinotopic maps in visual cortex van_essen_visual_1984schwartz_computational_1994. We set $a=0.5$ and a field-of-view of 16 degrees. B. The integral of the CMF $w$, from $0$ to $r$, yielding the "cortical" dimension corresponding to eccentricity. C. Sampling evenly along the domain of $w$ and solving for the corresponding retinal radius $r$ to achieve foveated samples in visual space. D. Sensor locations in visual space arising from isotropic foveated sampling. E. Visual points from D. mapped in the manifold of rovamo_isotropy_1984. F. Visual points from D. mapped into the complex log model schwartz_computational_1980.
• Figure 3: Kernel mapping procedure for the kNN-convolution operation. First, even k-sized neighborhoods are defined in the sensor manifold (upper), along with their orientation in the visual input (cartesian) space (lower). Second, kernels are transformed into a common cartesian reference frame, and aligned to a common reference kernel. Third, we visualize the learned reference kernel across different units in both the flattened representational space (top), and in visual space (bottom). These show how the kernel is a fixed size in the sensor manifold, and orientionally-aligned in visual cartesian coordinates, while scaling with eccentricity.
• Figure 4: FOVI-CNNs account for the spatial characteristics of primate neural receptive fields, and provide a performance advantage in ImageNet classification. A. Implementing variable foveation via variability in the cortical magnification function (CMF). Given the cortical magnification function (CMF; $M(r)$), we can specify a continuum of foveation, where small $a$ corresponds to strong foveation, and uniform sampling is achieved as $a\to \infty$. Note: we scale the CMF by $k_a$, where $k_a = (\int_0^{r_{\text{max}}} \frac{1}{r+a}\,dr)^{-1}$, in order to normalize the area-under-the-curve across models. B. Left: human population receptive field (pRF) sizes, measured with fMRI dumoulin_population_2008. Middle: receptive field (RF) sizes across layers in a foveated model ($a=0.5$). Right: RF sizes in a nearly uniformly sampling model $(a=50)$. C. Sampled fixations for strong and weak foveation models, shown both as a sensor array in visual space, and a (flat) sensor manifold in a V1-like space. D. ImageNet-1K results after training, mapping out performance up to 20 random fixations. E. Highlight of performance across foveation levels at the maximum of 20 fixations.
• Figure 5: Analyzing efficiency in ViT-S+ from the lens of foveation. A. GFLOPs/image as a function of image resolution, separately for attention and non-attention operations. Power laws are fit to each curve as empirical $O(m)$ analyses, where $m=\sqrt{n}$ is the pixels per side of a square image. B. GFLOPs/image as a function of the number of fixations, for different image resolutions. C. Local sampling resolution (native pixels per sensor sample) as a function of eccentricity, varying the resolution and foveation of the sensor. A horizontal line at 1 indicates the native resolution.