Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts

Shashank

TL;DR

The paper tackles catastrophic forgetting and inefficiency in Transformer FFNs by introducing Attractor Patch Networks (APN), a plug-in FFN replacement that routes tokens to a small, context-specific set of patch experts whose outputs are low-rank residuals. APN formalizes as a piecewise low-rank operator, with prototypes guiding routing and a compact code driving per-patch transformations, enabling localized plasticity and reduced interference. Empirical results on character-level Shakespeare modeling show competitive perplexity with the dense baseline while delivering dramatically better continual adaptation: retention improves by roughly 2.6× and adaptation by about 2.8× when moving from a base to a shifted domain. This architectural approach provides a scalable, interpretable inductive bias toward specialization and locality, suggesting broad potential for robust continual learning in language models without altering the core Transformer interface.

Abstract

Transformers achieve strong language modeling accuracy, yet their position-wise feed-forward networks (FFNs) are dense, globally shared, and typically updated end to end. These properties create two practical tensions. First, dense FFNs spend the same compute on every token regardless of context, and they allocate capacity uniformly even when language exhibits highly clustered context structure. Second, continual learning, in the sense of updating the model while serving a data stream, often produces interference because a small update touches broadly shared weights. We propose Attractor Patch Networks (APN), a plug-compatible replacement for the Transformer FFN. APN is a bank of patch experts. A similarity router selects a small top-k set of patches for each token by matching the token representation to learned prototypes. Each selected patch emits a low-rank residual update conditioned on a compact code. The architecture yields conditional, context-specialized nonlinear transformations while preserving the standard Transformer interface. This paper focuses on APN as an architectural primitive. We formalize APN, analyze its expressivity as a piecewise low-rank residual function class, and derive simple interference and stability arguments that make APN naturally compatible with continual learning. In experiments on character-level language modeling, APN achieves competitive perplexity (4.57 vs 4.32 PPL) while enabling dramatically better continual adaptation: when adapting to a shifted domain, APN achieves 2.6 times better retention (11.1 vs 29.4 PPL on the original domain) and 2.8 times better adaptation (6.4 vs 17.8 PPL on the new domain) compared to global fine-tuning of a dense FFN baseline.

Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts

TL;DR

The paper tackles catastrophic forgetting and inefficiency in Transformer FFNs by introducing Attractor Patch Networks (APN), a plug-in FFN replacement that routes tokens to a small, context-specific set of patch experts whose outputs are low-rank residuals. APN formalizes as a piecewise low-rank operator, with prototypes guiding routing and a compact code driving per-patch transformations, enabling localized plasticity and reduced interference. Empirical results on character-level Shakespeare modeling show competitive perplexity with the dense baseline while delivering dramatically better continual adaptation: retention improves by roughly 2.6× and adaptation by about 2.8× when moving from a base to a shifted domain. This architectural approach provides a scalable, interpretable inductive bias toward specialization and locality, suggesting broad potential for robust continual learning in language models without altering the core Transformer interface.

Abstract

Transformers achieve strong language modeling accuracy, yet their position-wise feed-forward networks (FFNs) are dense, globally shared, and typically updated end to end. These properties create two practical tensions. First, dense FFNs spend the same compute on every token regardless of context, and they allocate capacity uniformly even when language exhibits highly clustered context structure. Second, continual learning, in the sense of updating the model while serving a data stream, often produces interference because a small update touches broadly shared weights. We propose Attractor Patch Networks (APN), a plug-compatible replacement for the Transformer FFN. APN is a bank of patch experts. A similarity router selects a small top-k set of patches for each token by matching the token representation to learned prototypes. Each selected patch emits a low-rank residual update conditioned on a compact code. The architecture yields conditional, context-specialized nonlinear transformations while preserving the standard Transformer interface. This paper focuses on APN as an architectural primitive. We formalize APN, analyze its expressivity as a piecewise low-rank residual function class, and derive simple interference and stability arguments that make APN naturally compatible with continual learning. In experiments on character-level language modeling, APN achieves competitive perplexity (4.57 vs 4.32 PPL) while enabling dramatically better continual adaptation: when adapting to a shifted domain, APN achieves 2.6 times better retention (11.1 vs 29.4 PPL on the original domain) and 2.8 times better adaptation (6.4 vs 17.8 PPL on the new domain) compared to global fine-tuning of a dense FFN baseline.
Paper Structure (73 sections, 22 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 73 sections, 22 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background and Problem Setting
  4. Attractor Patch Networks
  5. Core definition
  6. Diagram
  7. Interpretation: attractors and specialization
  8. APN as a Function Class
  9. Piecewise structure
  10. Effective rank per token
  11. Why specialization can improve accuracy
  12. Capacity at fixed parameter budget
  13. Continual Learning Compatibility
  14. Patch locality and the update surface
  15. Interference proxy via patch overlap
  16. ...and 58 more sections

Figures (2)

  • Figure 1: Attractor Patch Network, a drop-in replacement for a Transformer FFN. Routing selects a small set of patch experts per token, each patch emits a low-rank residual update.
  • Figure 2: Continual learning intuition. If online updates touch only active patches, interference between two contexts is mediated by patch overlap.