Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Memorization: The Challenge of Random Memory Access in Language Models

Tongyao Zhu, Qian Liu, Liang Pang, Zhengbao Jiang, Min-Yen Kan, Min Lin

TL;DR

The paper investigates how decoder‑only language models store and access knowledge, revealing robust sequential memory recall but substantial challenges for random access to mid‑passage content. It introduces recitation and permutation as practical mitigations, showing improved performance on synthetic tasks and a meaningful boost in open‑domain QA when memory content is recited before querying. The findings highlight fundamental limits of parametric memory access and offer concrete strategies to load memory into the context, with implications for reliability and safety in knowledge‑dense applications. Overall, the work deepens understanding of memory access patterns in LMs and demonstrates that memory‑loading interventions can meaningfully enhance real‑world QA tasks.

Abstract

Recent developments in Language Models (LMs) have shown their effectiveness in NLP tasks, particularly in knowledge-intensive tasks. However, the mechanisms underlying knowledge storage and memory access within their parameters remain elusive. In this paper, we investigate whether a generative LM (e.g., GPT-2) is able to access its memory sequentially or randomly. Through carefully-designed synthetic tasks, covering the scenarios of full recitation, selective recitation and grounded question answering, we reveal that LMs manage to sequentially access their memory while encountering challenges in randomly accessing memorized content. We find that techniques including recitation and permutation improve the random memory access capability of LMs. Furthermore, by applying this intervention to realistic scenarios of open-domain question answering, we validate that enhancing random access by recitation leads to notable improvements in question answering. The code to reproduce our experiments can be found at https://github.com/sail-sg/lm-random-memory-access.

Beyond Memorization: The Challenge of Random Memory Access in Language Models

TL;DR

The paper investigates how decoder‑only language models store and access knowledge, revealing robust sequential memory recall but substantial challenges for random access to mid‑passage content. It introduces recitation and permutation as practical mitigations, showing improved performance on synthetic tasks and a meaningful boost in open‑domain QA when memory content is recited before querying. The findings highlight fundamental limits of parametric memory access and offer concrete strategies to load memory into the context, with implications for reliability and safety in knowledge‑dense applications. Overall, the work deepens understanding of memory access patterns in LMs and demonstrates that memory‑loading interventions can meaningfully enhance real‑world QA tasks.

Abstract

Recent developments in Language Models (LMs) have shown their effectiveness in NLP tasks, particularly in knowledge-intensive tasks. However, the mechanisms underlying knowledge storage and memory access within their parameters remain elusive. In this paper, we investigate whether a generative LM (e.g., GPT-2) is able to access its memory sequentially or randomly. Through carefully-designed synthetic tasks, covering the scenarios of full recitation, selective recitation and grounded question answering, we reveal that LMs manage to sequentially access their memory while encountering challenges in randomly accessing memorized content. We find that techniques including recitation and permutation improve the random memory access capability of LMs. Furthermore, by applying this intervention to realistic scenarios of open-domain question answering, we validate that enhancing random access by recitation leads to notable improvements in question answering. The code to reproduce our experiments can be found at https://github.com/sail-sg/lm-random-memory-access.
Paper Structure (39 sections, 4 figures, 11 tables)

This paper contains 39 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memorization in Language Models.
  4. Knowledge Injection.
  5. Knowledge Retrieval.
  6. Language Model as a Document Index.
  7. Investigating Sequential and Random Memory Access
  8. Task Formulation
  9. Sequential Access: Full Recitation
  10. Setup.
  11. Discussion.
  12. Random Access: Selective Recitation
  13. Setup.
  14. Discussion.
  15. Random Access: Grounded Question Answering
  16. ...and 24 more sections

Figures (4)

  • Figure 1: An illustration of our investigation of memory access pattern in language models. We find that the model accesses its parametric memory largely in a sequential manner, and faces difficulty in randomly accessing the content in the middle of memorized strings.
  • Figure 2: EM and BLEU for reading validation passages, varying the number of training passages. We calculate EM using only the first 25 tokens, as the model tends to continue generation beyond the max passage length (25).
  • Figure 3: A stacked bar plot showing the accuracy of ID-guided sentence recitation with different marker numbers. The performance decreases significantly as the sentence index grows, revealing the model's incapability in accessing middle sentences.
  • Figure 4: A illustration of the recitation method. The model first recites the corresponding passage content and subsequently extracts the answer in the context, in contrast to directly answering the question.