Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prometheus Mind: Retrofitting Memory to Frozen Language Models

Mark Wind

TL;DR

This work addresses the lack of persistent memory in large language models by proposing Prometheus Mind, a memory retrofit that attaches 11 small adapters to a frozen Qwen3-4B and injects memory through attention at layer 35, without modifying base weights. The approach hinges on four insights: extracting facts with Contrastive Direction Discovery (CDD), stage-wise adapter training to avoid end-to-end collapse, Identity V using lm_head rows as memory values to achieve zero-training generalization, and learned projections to repair hidden-state collapse. On PrometheusExtract-132, Prometheus Mind achieves 94.4% retrieval on clean inputs, 57.6% retrieval per-case, and 19.4% on adversarial inputs, with relation classification (47.3% accuracy) being the main bottleneck; overall, memory retrieval scales with category difficulty and memory size. The results demonstrate that memory can be effectively retrofitted to a frozen model via modular adapters, enabling reversible memory augmentation with modest overhead and without architectural changes, and they suggest broader implications for modular, composable cognition in AI systems.

Abstract

Adding memory to pretrained language models typically requires architectural changes or weight modification. We present Prometheus Mind, which retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead) -- fully reversible by removing the adapters. Building this system required solving four problems: (1) Extraction -- we develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data. (2) Training -- end-to-end optimization collapses; stage-wise training of each adapter on simple proxy tasks succeeds. (3) Injection -- learned encoders fail to generalize; we find that lm_head-weight rows already provide the mapping we need, requiring no training. (4) Hidden state collapse -- transformers make ``wife'' and ``brother'' 0.98+ similar; we train projections to recover distinction (0.98 $\rightarrow$ 0.09). On PrometheusExtract-132 (132 cases), the system achieves 94.4% retrieval on clean inputs (n=54, 95% CI: [84.9%, 98.1%]), degrading to 19.4% on informal inputs with ellipsis, filler words, or implicit subjects (n=36). The primary bottleneck is relation classification (47.3% accuracy), responsible for most extraction errors.

Prometheus Mind: Retrofitting Memory to Frozen Language Models

TL;DR

This work addresses the lack of persistent memory in large language models by proposing Prometheus Mind, a memory retrofit that attaches 11 small adapters to a frozen Qwen3-4B and injects memory through attention at layer 35, without modifying base weights. The approach hinges on four insights: extracting facts with Contrastive Direction Discovery (CDD), stage-wise adapter training to avoid end-to-end collapse, Identity V using lm_head rows as memory values to achieve zero-training generalization, and learned projections to repair hidden-state collapse. On PrometheusExtract-132, Prometheus Mind achieves 94.4% retrieval on clean inputs, 57.6% retrieval per-case, and 19.4% on adversarial inputs, with relation classification (47.3% accuracy) being the main bottleneck; overall, memory retrieval scales with category difficulty and memory size. The results demonstrate that memory can be effectively retrofitted to a frozen model via modular adapters, enabling reversible memory augmentation with modest overhead and without architectural changes, and they suggest broader implications for modular, composable cognition in AI systems.

Abstract

Adding memory to pretrained language models typically requires architectural changes or weight modification. We present Prometheus Mind, which retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead) -- fully reversible by removing the adapters. Building this system required solving four problems: (1) Extraction -- we develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data. (2) Training -- end-to-end optimization collapses; stage-wise training of each adapter on simple proxy tasks succeeds. (3) Injection -- learned encoders fail to generalize; we find that lm_head-weight rows already provide the mapping we need, requiring no training. (4) Hidden state collapse -- transformers make ``wife'' and ``brother'' 0.98+ similar; we train projections to recover distinction (0.98 0.09). On PrometheusExtract-132 (132 cases), the system achieves 94.4% retrieval on clean inputs (n=54, 95% CI: [84.9%, 98.1%]), degrading to 19.4% on informal inputs with ellipsis, filler words, or implicit subjects (n=36). The primary bottleneck is relation classification (47.3% accuracy), responsible for most extraction errors.
Paper Structure (86 sections, 8 equations, 1 figure, 12 tables)

This paper contains 86 sections, 8 equations, 1 figure, 12 tables.

Table of Contents

  1. Introduction
  2. The journey to memory.
  3. Limitations acknowledged upfront.
  4. Related Work
  5. Memory-Augmented Language Models
  6. Retrieval-Augmented Generation
  7. Adapter-Based Methods
  8. Activation Engineering and Steering Vectors
  9. Linear Probing and Semantic Subspaces
  10. Background
  11. Transformer Language Models
  12. Parameter-Efficient Adaptation
  13. Method
  14. System Overview
  15. Memory Structure.
  16. ...and 71 more sections

Figures (1)

  • Figure 1: Prometheus Mind architecture. Input flows through a frozen Qwen3-4B (left). CDD extracts facts into structured memory (right). At query time, multi-hop reasoning retrieves answers and injects them at layer 35 via attention K-V pairs, using Identity V (lm_head.weight).