PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis

TL;DR

This work proposes PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively, and reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.

Abstract

Predictive coding is a theory which hypothesises that cortex predicts sensory inputs at various levels of abstraction to minimise prediction errors. Inspired by predictive coding, Chen et al. (2024) proposed another theory, temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs. Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compensated by the following CA1 predictor. Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored. In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning. Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another. Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient. Building upon a popular non-contrastive learner, SimSiam, we propose PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively. Through studying the PhiNet model, we discover two findings. First, meaningful data representations emerge in PhiNet more stably than in SimSiam. This is initially supported by our learning dynamics analysis: PhiNet is more robust to the representational collapse. Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios. For practitioners, we additionally propose an extension called X-PhiNet integrated with a momentum encoder, excelling in continual learning. All in all, our work reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.

Figures (11)

• Figure 1: The architecture of SimSiam chen2021exploring and PhiNets. EMA in the X-PhiNet model stands for the exponential moving average. The architecture originates from a single input, branches out into three paths, and then compares the similarity of all paths in Sim-2. Thus, we call it PhiNet ($\Phi$-Net) because the shape of the architecture resembles the Greek letter Phi ($\Phi$).
• Figure 2: The interpretation as a hippocampal model. NC stands for NeoCortex.
• Figure 3: State space diagrams of PhiNet dynamics with different levels of weight decay: strong ($\rho=0.12$), medium ($\rho=0.03$), light ($\rho=0.003$), and weak ($\rho=0.0001$). The vector fields are numerically computed with $\sigma^2=1.5$. The state space bifurcates at the boundary of each level. The nullclines are shown with the green real ($\dot\psi=0$) and dotted ($\dot\gamma=0$) lines. The red dots are sinks.
• Figure 4: Illustration of SimSiam dynamics (\ref{['equation:ode_simsiam']}). Unlike the bivariate PhiNet dynamics shown in Figure \ref{['figure:vecfield_PhiNet']}, SimSiam dynamics is univariate, shown in the $\psi$-axis. The red dots are sinks.
• Figure 5: The SimSiam-medium flow is conjugate with the flow on the nullcline $\dot\psi=0$ (green real line) in PhiNet-medium.

Theorems & Definitions (2)

• Remark 1
• Lemma 1: tian2021understanding