Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformers are Stateless Differentiable Neural Computers

Bo Tang, Weiwei Xie

Abstract

Differentiable Neural Computers (DNCs) were introduced as recurrent architectures equipped with an addressable external memory supporting differentiable read and write operations. Transformers, in contrast, are nominally feedforward architectures based on multi-head self-attention. In this work we give a formal derivation showing that a causal Transformer layer is exactly a stateless Differentiable Neural Computer (sDNC) where (1) the controller has no recurrent internal state, (2) the external memory is a write-once matrix of value vectors, (3) content-based addressing via keys implements attention, and (4) multi-head attention corresponds to multiple parallel read heads. We further extend this equivalence to cross-attention, showing that encoder-decoder Transformers are precisely sDNCs with distinct read-from and write-to memories. Our results provide a unified memory-centric interpretation of Transformers and contribute to the ongoing effort to place modern large language models in a principled computational framework.

Transformers are Stateless Differentiable Neural Computers

Abstract

Differentiable Neural Computers (DNCs) were introduced as recurrent architectures equipped with an addressable external memory supporting differentiable read and write operations. Transformers, in contrast, are nominally feedforward architectures based on multi-head self-attention. In this work we give a formal derivation showing that a causal Transformer layer is exactly a stateless Differentiable Neural Computer (sDNC) where (1) the controller has no recurrent internal state, (2) the external memory is a write-once matrix of value vectors, (3) content-based addressing via keys implements attention, and (4) multi-head attention corresponds to multiple parallel read heads. We further extend this equivalence to cross-attention, showing that encoder-decoder Transformers are precisely sDNCs with distinct read-from and write-to memories. Our results provide a unified memory-centric interpretation of Transformers and contribute to the ongoing effort to place modern large language models in a principled computational framework.
Paper Structure (7 sections, 1 theorem, 11 equations, 1 figure)

This paper contains 7 sections, 1 theorem, 11 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Stateless Differentiable Neural Computers
  4. Main Result: Transformers are sDNCs
  5. Cross-Attention as Multi-Memory sDNC
  6. Discussion
  7. Conclusion

Key Result

Theorem 1

Consider a single causal Transformer self-attention layer with parameters $(W_Q,W_K,W_V,W_O)$ acting on a sequence $\mathbf{X} = (\mathbf{x}_1,\dots,\mathbf{x}_T)$. Let $\mathbf{z}_t$ denote its attention output at position $t$. Then there exists an sDNC such that its read vector $\mathbf{r}_t$ at t

Figures (1)

  • Figure 1: Diagram showing the structural equivalence between a Transformer layer and a stateless Differentiable Neural Computer (sDNC). A Transformer’s linear projections produce keys and values analogous to the sDNC controller’s emissions. The self-attention mechanism corresponds exactly to content-based reads from an external memory composed of value vectors. Transformer outputs match sDNC readouts for each position.

Theorems & Definitions (2)

  • Theorem 1
  • proof