Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling Language as a Sequence of Thoughts

Nasim Borazjanizadeh, James McClelland

TL;DR

Thought Gestalt (TG) introduces a two-level language model that couples token-level generation with sentence-level gestalts stored in differentiable memory, enabling gradients to backpropagate through memory to refine earlier sentence representations. By processing text sentence-by-sentence and cross-attending to a memory of prior sentence states, TG achieves higher data and parameter efficiency than a matched GPT-2 baseline and improves robustness on relational reasoning tasks such as the reversal curse. Key findings show TG attains 2–4% lower perplexity across data scales and implies a data-efficiency gain of about 5–8% and a parameter-efficiency gain of roughly 1.3–1.4× to match TG’s loss, while ablations confirm the essential role of gradient flow through memory and the benefits of semantically coherent sentence memory. The approach advances the goal of learning latent situation models and robust abstractions by integrating memory-driven sentence gestalts, with potential implications for scalable, reasoning-capable language systems and hierarchical representations in future work.

Abstract

Transformer language models can generate strikingly natural text by modeling language as a sequence of tokens. Yet, by relying primarily on surface-level co-occurrence statistics, they fail to form globally consistent latent representations of entities and events, lack of which contributes to brittleness in relational direction (e.g., reversal curse), contextualization errors, and data inefficiency. On the other hand, cognitive science shows that human comprehension involves converting the input linguistic stream into compact, event-like representations that persist in memory while verbatim form is short-lived. Motivated by this view, we introduce Thought Gestalt (TG) model, a recurrent Transformer that models language at two levels of abstraction - tokens and sentence-level "thought" states. TG generates the tokens of one sentence at a time while cross-attending to a memory of prior sentence representations. In TG, token and sentence representations are generated using the same set of model parameters and trained with a single objective, the next-token cross-entropy: by retaining the computation graph of sentence representations written to memory, gradients from future token losses flow backward through cross-attention to optimize the parameters generating earlier sentence vectors. In scaling experiments, TG consistently improves efficiency over matched GPT-2 runs, among other baselines, with scaling fits indicating GPT-2 requires ~5-8% more data and ~33-42% more parameters to match TG's loss. TG also reduces errors on relational direction generalization on a father-son reversal curse probe.

Modeling Language as a Sequence of Thoughts

TL;DR

Thought Gestalt (TG) introduces a two-level language model that couples token-level generation with sentence-level gestalts stored in differentiable memory, enabling gradients to backpropagate through memory to refine earlier sentence representations. By processing text sentence-by-sentence and cross-attending to a memory of prior sentence states, TG achieves higher data and parameter efficiency than a matched GPT-2 baseline and improves robustness on relational reasoning tasks such as the reversal curse. Key findings show TG attains 2–4% lower perplexity across data scales and implies a data-efficiency gain of about 5–8% and a parameter-efficiency gain of roughly 1.3–1.4× to match TG’s loss, while ablations confirm the essential role of gradient flow through memory and the benefits of semantically coherent sentence memory. The approach advances the goal of learning latent situation models and robust abstractions by integrating memory-driven sentence gestalts, with potential implications for scalable, reasoning-capable language systems and hierarchical representations in future work.

Abstract

Transformer language models can generate strikingly natural text by modeling language as a sequence of tokens. Yet, by relying primarily on surface-level co-occurrence statistics, they fail to form globally consistent latent representations of entities and events, lack of which contributes to brittleness in relational direction (e.g., reversal curse), contextualization errors, and data inefficiency. On the other hand, cognitive science shows that human comprehension involves converting the input linguistic stream into compact, event-like representations that persist in memory while verbatim form is short-lived. Motivated by this view, we introduce Thought Gestalt (TG) model, a recurrent Transformer that models language at two levels of abstraction - tokens and sentence-level "thought" states. TG generates the tokens of one sentence at a time while cross-attending to a memory of prior sentence representations. In TG, token and sentence representations are generated using the same set of model parameters and trained with a single objective, the next-token cross-entropy: by retaining the computation graph of sentence representations written to memory, gradients from future token losses flow backward through cross-attention to optimize the parameters generating earlier sentence vectors. In scaling experiments, TG consistently improves efficiency over matched GPT-2 runs, among other baselines, with scaling fits indicating GPT-2 requires ~5-8% more data and ~33-42% more parameters to match TG's loss. TG also reduces errors on relational direction generalization on a father-son reversal curse probe.
Paper Structure (48 sections, 2 equations, 6 figures, 5 tables)

This paper contains 48 sections, 2 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Other Works
  3. Learning sentence representations.
  4. Sequence compression and gisting.
  5. Recurrence and memory in Transformers.
  6. Main Design Principles
  7. Data Preparation
  8. Sentence splitting.
  9. Sentence tokenization.
  10. Sentence streams as training examples and batch construction.
  11. Model
  12. Sentence representation and memory.
  13. Cross-attention to sentence memory.
  14. Token embeddings and self-attention.
  15. Learnable memory gates.
  16. ...and 33 more sections

Figures (6)

  • Figure 1: The Thought Gestalt (TG) model learns token and sentence representations jointly using a single next-token prediction objective; by retaining the computation graph at memory writes, token prediction loss from future sentences backpropagates through cross-attention to refine the parameters that generated earlier sentence representations.
  • Figure 2: Kaplan-style scaling behavior. Panel (a) varies training data size $D$ at fixed model size of 85M non-embedding parameters for TG and GPT-2; panel (b) varies model size $N$ at fixed training dataset size of 50M tokens.
  • Figure 3: TG model architecture. Alternating self-attention and cross-attention blocks compose the main model stack, which predicts tokens of one sentence at a time. EOS at layer 7 is projected by a small linear “sentence head’’ to produce the sentence vector. Computation graph of sentence vectors are retained on memory writes; memory keys/values are cached.
  • Figure 4: Father--son reversal-curse: We report mean NLL (lower is better) of the gold single-token answer under normal (direction-matched) and reversed (direction-inverted) queries. Markers distinguish TG (circles) from GPT-2 baselines (squares); filled vs. hollow markers indicate whether the top-1 prediction matches the gold answer. Pretrained GPT-2 and Mistral 7B are included as web-scale reference points.
  • Figure 5:
  • ...and 1 more figures