Recursive Dynamics in Fast-Weights Homeostatic Reentry Networks: Toward Reflective Intelligence

TL;DR

The paper introduces the Fast-Weights Homeostatic Reentry Layer (FH-RL), a minimal Transformer extension that enables internal recursive self-reference by integrating fast-weight associative memory, homeostatic normalization, and learned reentrant feedback. It defines three quantitative metrics—Information Reentry Ratio ($\mathrm{IRR}$), Eigen-Spectrum Recursion Index ($\mathrm{ESRI}$), and Representational Drift Periodicity ($\mathrm{RDP}$)—to capture self-referential dynamics and demonstrates a stable reflective band around $\gamma \in [0.10,0.20]$ where internal feedback is expressive yet spectrally stable. Experiments on a toy-scale Tiny-GPT model with a byte-level synthetic dataset show that increasing reentry strength boosts IRR while keeping ESRI near zero and maintaining a fixed RDP around $f \approx 0.225$, with Wr forming high-dimensional but structured, non-token-aligned feedback. The findings provide quantitative evidence that reflective, thought-like processing can emerge from a principled balance between feedback amplification and homeostatic regulation, linking fast-weight architectures to cortical reentry concepts and suggesting a controllable axis for artificial metacognition.

Abstract

This study introduces the Fast-Weights Homeostatic Reentry Layer (FH-RL), a neural mechanism that integrates fast-weight associative memory, homeostatic regularization, and learned reentrant feedback to approximate self-referential computation in neural networks. Unlike standard transformer architectures that operate in a purely feedforward manner during inference, FH-RL enables internal recurrence without external looping, allowing prior latent states to be dynamically re-entered into the ongoing computation stream. We conduct controlled experiments sweeping the reentry gain $γ$ and evaluate emergent internal dynamics using three novel metrics: the Information Reentry Ratio (IRR), Eigen-Spectrum Recursion Index (ESRI), and Representational Drift Periodicity (RDP). Results show that reentry quantity increases proportionally with~$γ$, while the learned feedback matrix $W_r$ remains bounded and becomes more structured at moderate gains. Critically, a stable reflective band emerges around $γ\approx 0.10-0.20$, where internal feedback is maximally expressive yet spectrally stable: IRR rises smoothly, ESRI remains near zero, and RDP exhibits consistent low-frequency cycles. These findings provide quantitative evidence that reflective, thought-like internal processing can arise from a principled balance between feedback amplification and homeostatic regulation, linking modern fast-weight architectures to theories of cortical reentry and recursive cognition.

Figures (8)

• Figure 1: Conceptual reentrant loop in the FH-RL model using fast-weights and learned feedback projection.
• Figure 2: Training loss across reentry strengths $\gamma \in [0.0,\,0.30]$.
• Figure 3: Figure 3. Spectral and energetic signatures of reentrant feedback as a function of reentry strength $\gamma$. The blue axis (left) indicates the ratio of the information reentry rate with feedback to the effective spectral reentry index [IRR(with $\gamma$)/ESRI]. The green axis (right) represents the information reentry rate computed using only the recurrent weight component IRR ($W_r$ only).
• Figure 4: Stable representational drift periodicity across reentry strengths.
• Figure 5: Heatmaps of the reentry projection matrix $W_r$ for different $\gamma \in \{0.0, 0.1, 0.2, 0.3\}$. Visual patterns are largely indistinguishable across $\gamma$, indicating that feedback organization is high-dimensional and distributed.