RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers

TL;DR

RMAAT tackles long-context sequence modeling by integrating astrocyte-inspired memory and attention into a recurrent, segment-based Transformer. It introduces a macro model of neuron–astrocyte LTP/LTP dynamics to derive a memory retention factor for adaptive compression of context between segments, and AMRB to train efficiently without full backpropagation through all tokens. The astromorphic attention achieves $O(N)$ complexity per segment using fixed-size intermediates and a Hebbian-like mechanism modulated by astrocytes, with STP-grounded relative positional encoding enhancing temporal-spatial context. Experiments on Long Range Arena show competitive accuracy with substantially lower memory usage and faster training than recurrent baselines, demonstrating the potential of neuro-glial principles for scalable long-range sequence processing.

Abstract

The quadratic complexity of self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. This work explores computational principles derived from astrocytes-glial cells critical for biological memory and synaptic modulation-as a complementary approach to conventional architectural modifications for efficient self-attention. We introduce the Recurrent Memory Augmented Astromorphic Transformer (RMAAT), an architecture integrating abstracted astrocyte functionalities. RMAAT employs a recurrent, segment-based processing strategy where persistent memory tokens propagate contextual information. An adaptive compression mechanism, governed by a novel retention factor derived from simulated astrocyte long-term plasticity (LTP), modulates these tokens. Attention within segments utilizes an efficient, linear-complexity mechanism inspired by astrocyte short-term plasticity (STP). Training is performed using Astrocytic Memory Replay Backpropagation (AMRB), a novel algorithm designed for memory efficiency in recurrent networks. Evaluations on the Long Range Arena (LRA) benchmark demonstrate RMAAT's competitive accuracy and substantial improvements in computational and memory efficiency, indicating the potential of incorporating astrocyte-inspired dynamics into scalable sequence models.

Figures (9)

• Figure 1: Conceptual illustration of RMAAT processing through time unrolling. Processing within each segment incorporates mechanisms inspired by STP. The recurrent propagation of astrocytic memory tokens ($mem_t$) integrates context across many segments, drawing inspiration from LTP principles for persistent memory.
• Figure 2: Overview of the Astromorphic Transformer architecture. This diagram illustrates the integration of bioplausible bidirectional feedback mechanisms within a two-layered neuron-astrocyte network, emulating the Self-Attention of the transformer encoder. The synaptic weights $W_{K}$, $W_{Q}$, and $W_{V}$ are corresponding to key ($K$), query ($Q$) and value ($V$) of the transformer which are activated once the input ($X$) is received. Write Mode (marked in brown): Hebbian plasticity ($H$) between the pre and postneurons is denoted by $H_{neuron}$ and that between the postneuron and astrocyte is denoted by $H_{astro}$ (dashed lines represent the bidirectional connections among the elements of the tripartite synapse). $H_{astro}$ calculation involves the astrocytic parameter, $R$ which is computed based on the spatial positions of neurons quantified by $T_{ijkl}$. The calcium response of the astrocyte ($C$) is encoded from the $Ca^{2+}$ concentration in the astrocyte evoked by the presynaptic action potential ($K$). Read Mode (marked in red): Presynaptic plasticity ($P$) is decoded by the presynaptic action potential ($Q$) and the final weight $H\odot P$ defines the synaptic weight between the hidden ($h$) and the output layer ($L$).
• Figure 3: Simulation of the computational neuroscience model ($3\times3$-neuron network ($9$ connections), $300$s total time, $6\times50$s STP cycles: STP cycles are reset every $50$s in the $300$s simulation) illustrating temporal integration for astrocyte-inspired memory. Dashed lines show the long-term astrocyte process ($p^l_{ij}$) integrating information and gradually saturating across STP cycles. Solid lines show the faster synaptic facilitation dynamics ($s_{ij}$) within each STP cycle.
• Figure 4: Memory Retention Factor derived from simulating the LTP macro model for different total sequence lengths (represented as total number of STP cycles from $2$ to $8$). The factor decreases per segment as the total sequence length increases, implementing adaptive, bio-inspired context compression.
• Figure S1: Spatial layout of the $N=5$ neurons used in the simulation.