Semantic representations emerge in biologically inspired ensembles of cross-supervising neural networks

TL;DR

The study investigates how brains might form semantic representations under weak supervision by using ensembles of partially driven networks that cross-supervise each other. It introduces CLoSeR, a cross-supervised learning framework with encoder-specific fixed input masks and no weight sharing, extended to temporal data as TCLoSeR. Across vision benchmarks and neural population data, CLoSeR yields semantically meaningful embeddings that are easily decodable and competitive with supervised baselines, with optimal performance at small receptive fields and with sparse, structured connectivity. The temporal extension demonstrates that cross-supervision combined with temporal contiguity can produce robust representations in neural data, proposing a biologically plausible route to distributed learning and collective computation in the brain.

Abstract

Brains learn to represent information from a large set of stimuli, typically by weak supervision. Unsupervised learning is therefore a natural approach for exploring the design of biological neural networks and their computations. Accordingly, redundancy reduction has been suggested as a prominent design principle of neural encoding, but its ``mechanistic'' biological implementation is unclear. Analogously, unsupervised training of artificial neural networks yields internal representations that allow for accurate stimulus classification or decoding, but typically rely on biologically-implausible implementations. We suggest that interactions between parallel subnetworks in the brain may underlie such learning: we present a model of representation learning by ensembles of neural networks, where each network learns to encode stimuli into an abstract representation space by cross-supervising interactions with other networks, for inputs they receive simultaneously or in close temporal proximity. Aiming for biological plausibility, each network has a small ``receptive field'', thus receiving a fixed part of the external input, and the networks do not share weights. We find that for different types of network architectures, and for both visual or neuronal stimuli, these cross-supervising networks learn semantic representations that are easily decodable and that decoding accuracy is comparable to supervised networks -- both at the level of single networks and the ensemble. We further show that performance is optimal for small receptive fields, and that sparse connectivity between networks is nearly as accurate as all-to-all interactions, with far fewer computations. We thus suggest a sparsely interacting collective of cross-supervising networks as an algorithmic framework for representational learning and collective computation in the brain.

Figures (10)

• Figure 1: Schematic illustration of the CLoSeR framework.(a) Given an input (e.g., an image), each encoder receives a subset of the different input channels or pixels in images. The masking is randomly chosen for each network before training and remains fixed. Each encoder maps stimuli into some "embedding space". In this example, masking is sampled with a spatial Gaussian. (b) A schematic illustration of the training loss: For every pair of encoders $\mu_i$ and $\mu_j$, the encoders train to minimize the distance over their representations of the same stimulus (green arrows), and maximize distance of different ones (red arrows).
• Figure 2: CLoSeR with local Gaussian masking learns semantically meaningful representations.(a) left: The average value of the conditional pseudo-likelihood of an image from one Cifar-10 class given an image from another class, for one untrained MLP encoder that receives 10% of the pixels of the image. Center: same for an encoder that was trained using CLoSeR. Right: Same for an encoder that was trained in a supervised manner. All panels show results for an encoder that was initialized the same way. (b) Left: The distribution over 10 sets of mean Cifar-10 accuracy values of a linear classifier decoding of each encoder's embedding for the case of CLoSer trained MLP encoders (green) untrained encoders (orange), or encoders trained by supervision (pink). The encoders in this figure receive 10% of the pixels of the input images. Right: The distribution of accuracy values of a linear classifier decoding from the collective embeddings of the ensemble. The dotted black line marks chance level (10%). (c) Same as b for model trained and evaluated on Cifar-100. The dotted black line marks chance level (1%). (d) Accuracy of linear classifier decoding of a single encoding network as a function the number of encoding networks that were trained together, with 10% of the image presented to each encoder. The 95% CI was calculated over 10 seeds. The colors are the same as b and c, except for the blue, which is the accuracy when training a linear classifier on the masked images pixels.
• Figure 3: Performance of CLoSeR encoders is optimal for small input masking fraction, for both Gaussian and randomly dispersed receptive fields.(a) Example of the Gaussian masking method and how it changes as an encoder receives more pixels as input. (b) The average accuracy of a supervised linear classifier decoding the 10 MLP encoders with Gaussian masking as a function of the fraction of the image each encoder receives as input. Each line shows results for a different encoder class, and the shading shows the 95% confidence interval over different masking seeds ($n=10$). (c) Same as b for ensemble accuracy as a function of the fraction of image each individual encoder receives as input. (d) Example of the random patches masking method and how it changes when an encoder receives more patches as input. (e-f) Same as b and c for random patches masking.
• Figure 4: CLoSeR with sparse interactions is highly efficient:(a) Different interaction graphs we examined. Top row: illustration of the interaction graph. Bottom row: adjacency matrices of each of the interaction graphs depicted above. (b) The accuracy of single MLP Gaussian masked encoder classification as a function of the density of the interaction graph $|E|/N\cdot(N-1)$. Different colors correspond to different graph families, as in panel a. The encoders receive 10% of the image pixels as input. The directional ring and bidirectional ring single encoder accuracies are higher than $G_N^k$ (paired student t-test, $p\approx4.3\cdot10^{-3}$ and $p\approx8.9\cdot10^{-3}$, $df=9$). * designates $p<0.05$ and ** designates $p<0.01$. 95% CI is calculated over 10 seeds. (c) Same as b for ensemble accuracy, with $p\approx4.4\cdot10^{-2}$ and $p\approx4.8\cdot10^{-3}$.
• Figure 5: Visual classification accuracy for Vision Transformer trained with CLoSeR:(a) The Vision Transformer architecture, taken from vit. (b) Up: example of the contiguous patches masking method, for 20% of the image pixels. (c) Single encoder's linear classifier accuracy as a function of the image fraction seen by each ViT encoder with contiguous patches. Each line corresponds to a different training of networks. The 95% CI was calculated over 10 seeds. (d) Same as c for ensemble accuracy. (e) Example of the random patches masking method for 20% of the image pixels. (f-g) Same as c-d, for random patches masking.