Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

Ying Xie

Abstract

Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.

Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

Abstract

Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.
Paper Structure (67 sections, 3 theorems, 25 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 67 sections, 3 theorems, 25 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The Biological Precedent: Sleep as Active Forgetting
  3. Our Proposal
  4. Related Work
  5. Context Window Limitations and the Lost-in-the-Middle Effect.
  6. Proactive Interference in Human Cognition.
  7. Proactive Interference in LLMs.
  8. KV Cache Optimization.
  9. Memory-Augmented Architectures.
  10. Biological Inspiration in Machine Learning.
  11. Forgetting Mechanisms.
  12. The SleepGate Framework
  13. Preliminaries and Notation
  14. Module 1: Conflict-Aware Temporal Tagger
  15. Semantic Signatures.
  16. ...and 52 more sections

Key Result

Theorem 1

Under the assumptions that (i) the forgetting gate identifies superseded entries with probability $p_c \geq 1 - \epsilon$ for $\epsilon < 1$, and (ii) sleep cycles occur at intervals of $N$ tokens, the expected number of stale entries competing with the current value after $n$ updates is bounded by: For $p_c$ close to 1 (low $\epsilon$) and moderate $N$, this is $O(N)$ regardless of $n$---a consta

Figures (2)

  • Figure 1: Simplified overview of SleepGate. During the wake phase, the model processes input tokens while maintaining a tagged KV cache. A sleep trigger decides whether to continue normal inference or enter a sleep micro-cycle. In the soft biasing variant used in our experiments, the cache passes through decay and the forgetting gate, and the resulting retention scores are converted to additive attention biases (Eq. \ref{['eq:soft_bias']}) for a second forward pass. In the hard eviction variant, entries additionally pass through consolidation before eviction.
  • Figure 2: Retrieval accuracy vs. PI depth (number of prior updates). SleepGate (solid red) maintains near-perfect accuracy through $n = 10$ before degrading at higher depths. All baselines cluster near zero across all depths.

Theorems & Definitions (4)

  • Theorem 1: Interference Reduction
  • proof : Proof sketch
  • Corollary 1
  • Proposition 1: Cache Size Bound