Modeling Rapid Contextual Learning in the Visual Cortex with Fast-Weight Deep Autoencoder Networks

TL;DR

This paper investigates how rapid global contextual learning can emerge in early visual representations by combining a ViT-based autoencoder with LoRA-based fast weights. It demonstrates that familiarity training compresses nuisance variation and aligns early-layer representations with a global context stored in the top layer, with LoRA amplifying these effects and enriching self-attention with global context while preserving task flexibility. The results support a slow-core/fast-shell view of cortical-like plasticity, showing that fast, low-rank adapters can model rapid, context-dependent updates in hierarchical networks. These findings have implications for understanding brain-like rapid adaptation and for designing adaptive, context-aware vision systems.

Abstract

Recent neurophysiological studies have revealed that the early visual cortex can rapidly learn global image context, as evidenced by a sparsification of population responses and a reduction in mean activity when exposed to familiar versus novel image contexts. This phenomenon has been attributed primarily to local recurrent interactions, rather than changes in feedforward or feedback pathways, supported by both empirical findings and circuit-level modeling. Recurrent neural circuits capable of simulating these effects have been shown to reshape the geometry of neural manifolds, enhancing robustness and invariance to irrelevant variations. In this study, we employ a Vision Transformer (ViT)-based autoencoder to investigate, from a functional perspective, how familiarity training can induce sensitivity to global context in the early layers of a deep neural network. We hypothesize that rapid learning operates via fast weights, which encode transient or short-term memory traces, and we explore the use of Low-Rank Adaptation (LoRA) to implement such fast weights within each Transformer layer. Our results show that (1) The proposed ViT-based autoencoder's self-attention circuit performs a manifold transform similar to a neural circuit model of the familiarity effect. (2) Familiarity training aligns latent representations in early layers with those in the top layer that contains global context information. (3) Familiarity training broadens the self-attention scope within the remembered image context. (4) These effects are significantly amplified by LoRA-based fast weights. Together, these findings suggest that familiarity training introduces global sensitivity to earlier layers in a hierarchical network, and that a hybrid fast-and-slow weight architecture may provide a viable computational model for studying rapid global context learning in the brain.

Figures (6)

• Figure 1: (a) Neural Recurrent Circuit Model for Familiarity training (see Wang et al (2025)) (b) Familiarity Suppression -- population averaged responses to familiar images were found to be suppressed relative to their averaged responses to novel images. (see Huang et al. (2018)) (c) Sharpening of the population response to familiar images relative to novel images suggests the neural representation of familiar images becomes sparsified. (d) ViT-based Autoencoder Framework for modeling familiar training using deep networks. (e) Testing and Evaluation Framework: The network was trained with clean images but tested with noisy images with different levels of noise. Four types of evaluations were performed to compare with the recurrent circuit, to assess the impact of familiarity training on manifold transform, development of global context sensitivity in early layers of the Image Encoder, an analog of the early visual cortex, and the modification of attention focus in the self-attention map.
• Figure 2: (a) Illustration of the manifold geometry for two image contexts and the associated noise-cone. The cone’s cross-section ellipses show sample variability at each noise level, and its axis marks increasing noise. Distances across noise levels and within each level quantify how the manifold expands or contracts. (b) Learning curve of relative distances in the recurrent neural circuit model. (c-e) ViT encoder with LoRA. (f-h) ViT encoder without LoRA. Training reduces both relative level and residual distances, indicating manifold compression.
• Figure 3: Evolution of the classification accuracy of the linear decoder applied to every transformer layer output during familiar training. The test stimuli were the different image contexts remembered corrupted with 30 % salt and pepper noises. Decoding accuracy reflects the average of the discriminability or untangling of any two trained image contexts. Networks with LoRA fast weights ( b) exhibit stronger improvement of discriminability than networks without LoRA ( a).
• Figure 4: Cosine similarity between the class token embedding of every layer and the top (i.e. the 11th) layer of the Image Encoder which reflects global context. LoRA network's layers consistently experience a progressive increase in the alignment with the global context, starting in layer 1. This alignment is considerably weaker for networks without LoRA, particularly in the early layers. The increase in alignment with the global context seems to be stabilized by 20 epochs and is essentially the same regardless of noise level.
• Figure 5: Layer-wise attention maps before and after familiarity training at 0% (clean) and 30 % noisy levels. Before familiar training, the attention map highlights accurately the critical features for object recognition a, but becomes scattered when the input stimulus is noisy b. Familiarity training makes the network pay attention to a large portion of the image, reflecting the encoding of more details and a broader scope of the image for remembering the entire image context, at the expense of focused attention to critical details for recognition. LoRA makes the expansion of attention more prominent ( e vs c). On the other hand, attention maps become more robust and stable against noise than before familiar training. ( c,d vs e,f ) respectively.