Mind the Gap: Why Neural Memory Fails Under Semantic Density

TL;DR

The paper identifies a fundamental limitation in online neural memory: when facts are semantically dense, storing them in shared continuous parameters leads to interference and rapid memory collapse, a phenomenon termed the Stability Gap governed by the Orthogonality Constraint. It argues that fast-weight or context-based memories cannot reliably store precise episodic facts under realistic semantic density, and production systems have already adopted discrete storage but suffer from schema drift and version ambiguity. The authors propose Knowledge Objects KO as discrete, typed memory units with version chains and controlled vocabularies, paired with neural weights for slow semantic generalization, and a learned router to direct factual versus fuzzy queries, forming a bicameral architecture that promises more reliable, scalable AI memory. The work also provides extensive empirical evidence across multiple experiments showing KO’s deterministic overwrite, surgical correction, and cost advantages over context-based memory, while exposing the limitations of RAG and pure neural approaches in dynamic memory tasks. Overall, the paper argues for a principled separation of memory modalities to achieve trustworthy, efficient AI memory suitable for production deployment.

Abstract

The brain solves a problem that current AI architectures struggle to manage: storing specific episodic facts without corrupting general semantic knowledge. Neuroscience explains this through Complementary Learning Systems theory - a fast hippocampal system for episodic storage using pattern-separated representations, and a slow neocortical system for extracting statistical regularities. Current AI systems lack this separation, attempting both functions through neural weights alone. We identify the 'Stability Gap' in online neural memory: fast-weight mechanisms that write facts into shared continuous parameters collapse to near-random accuracy within tens of semantically related facts. Through semantic density (rho), we show collapse occurs with as few as N=5 facts at high density (rho > 0.6) or N ~ 20-75 at moderate density - a phenomenon we formalise as the Orthogonality Constraint. This failure persists even with perfect attention and unlimited context, arising from write-time interference when storage and retrieval share the same substrate. We also identify schema drift and version ambiguity as primary failure modes in production systems, observing 40-70% schema consistency and 0-100% clean correction rates. Context-based memory incurs 30-300% cost premium over selective retrieval. We propose Knowledge Objects (KOs): discrete, typed memory units with controlled vocabularies and explicit version chains. Paired with neural weights, KOs enable a true complementary learning architecture, suggesting reliable AI memory may require this bicameral design.

Figures (5)

• Figure 1: The Memory Architecture Gap.(a) Pure neural approaches to online memory---context windows and fast weights---face the Orthogonality Constraint: interference grows with semantic density, causing collapse. Production systems have adopted discrete external storage (databases) but suffer from schema drift and version ambiguity. (b) Our hybrid architecture provides structured discrete storage (Knowledge Objects) with controlled vocabularies and version chains, connected to the LLM via a learned router. This explains why discrete storage is necessary and fixes the remaining failure modes.
• Figure 2: The Orthogonality Constraint.Left: In neural memory, semantically related facts (here, two statements about French cities) produce embedding vectors with small angular separation ($\cos\theta \approx 0.7$). When retrieving one fact, the system also partially retrieves others with similar keys---interference that grows with corpus size $N$ and semantic density $\rho$, causing collapse by $N \approx 20$--$75$ depending on density. Right: Discrete memory assigns unique hash addresses to each fact; overlap between addresses is exactly zero by construction. Retrieval returns $\mathbf{v}_A$ with no interference regardless of how many other facts share semantic similarity.
• Figure 3: Hybrid Architecture. Queries are embedded and classified by a Learned Router. Factual queries retrieve from the KO hash map; fuzzy queries pass directly to the LLM. This division of labor---discrete memory for facts, neural weights for generation---is the core architectural principle.
• Figure 4: The Stability Gap. Neural memory (red) collapses as $N$ increases, while discrete memory (green) maintains 100% accuracy throughout. Embedding dimension $d=384$ is held constant; only the number of stored facts varies. The shaded region represents the accuracy gap caused by write-time interference in continuous vector storage. Even as we store more facts, no improvement in attention or retrieval mechanism can prevent this collapse---it is a geometric consequence of semantic overlap in shared parameters.
• Figure 5: Semantic Density Predicts Collapse.$N_{50}$ (the number of facts at which neural accuracy drops to 50%) decreases sharply as semantic density ($\rho$) increases. The exponential decay validates the Orthogonality Constraint: higher $\rho$ means more key overlap, hence more interference. Note: $\rho < 0.3$ was not achievable with realistic fact structures.