Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

TL;DR

Memoria tackles the challenge of fateful forgetting in long-horizon sequence processing by introducing a human-inspired external memory that dynamically manages working, short-term, and long-term engrams. It couples a memory-graph–based retrieval mechanism with a memory encoder and cross-attention to enable selective preservation, cue-based activation, and iterative memory searching, implemented in Memoria Transformer and Memoria BERT. Across sorting, language modeling, and long-document classification, Memoria demonstrates superior retention of long-range information and reveals human-like memory effects such as primacy, recency, and temporal contiguity in its memory dynamics. The work offers a practical pathway to integrate memory-rich architectures into modern transformers and outlines future directions, including deeper cognitive mechanisms and privacy considerations for long-term memory in AI systems.

Abstract

Making neural networks remember over the long term has been a longstanding issue. Although several external memory techniques have been introduced, most focus on retaining recent information in the short term. Regardless of its importance, information tends to be fatefully forgotten over time. We present Memoria, a memory system for artificial neural networks, drawing inspiration from humans and applying various neuroscientific and psychological theories. The experimental results prove the effectiveness of Memoria in the diverse tasks of sorting, language modeling, and classification, surpassing conventional techniques. Engram analysis reveals that Memoria exhibits the primacy, recency, and temporal contiguity effects which are characteristics of human memory.

Figures (15)

• Figure 1: Working memory retains the most recent information. Short-term memory also holds a fixed number of recent engrams. The number of engrams in long-term memory is not predetermined. The arrows in the diagram represent the connections between each engram. Each connection is directed and weighted. Those weighted edges are used for retrieval.
• Figure 2: Retrieval process in Memoria. Memoria utilizes working memory to identify associated engrams in both short-term and long-term memory. The calculated weights in steps 1 and 4 mean the strength of association between the engrams and working memory, with larger values leading to the final selection of the engram. This mechanism deals with the cue-based activation problem by reflecting the association with working memory. Engrams in the gray area represent retrieved engrams.
• Figure 3: The connections between working memory and retrieved engrams are strengthened across all pairs. The lifespan of retrieved engrams extends proportionally to their individual contribution, enabling selective preservation through the differential allocation of lifespans. Engrams having lost all their lifespan, exemplified by $e_{ltm,7}$, are eliminated permanently.
• Figure 4: Results of sorting task. Memoria Transformer exhibits greater robustness compared to other baselines as the input sequence length increases. This task requires the retention of information about the occurrence of initial tokens until the end. While the other methods all show significant performance decline, Memoria Transformer successfully handles the issue of fateful forgetting, setting it apart from other competing techniques. The comprehensive raw scores are specified in \ref{['table:sorting-full-result']}.
• Figure 5: A structural diagram of Memoria Transformers.