Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

Zuher Jahshan, Ben Ben Ishay, Leonid Yavits

Abstract

MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.

MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

Abstract

MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.
Paper Structure (18 sections, 6 equations, 5 figures, 3 tables)

This paper contains 18 sections, 6 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and background
  3. Concept
  4. Contextualization
  5. Multi-head attention as contextualization
  6. MANAR
  7. Notations
  8. ACR construction and Token Contextualization
  9. The Memory Unit
  10. Multi-head architecture
  11. Evaluation
  12. Language Modeling
  13. Image Classification
  14. Knowledge Transfer
  15. Performance Comparison
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Geometric interpretation of contextualization with and without retrieved memory concepts.
  • Figure 2: High level architecture of MANAR
  • Figure 3: Comparing the time and space complexity of MANAR against vanilla MHA (left), and MANAR-based DeiT-S (MANAR-256.32.96-S) against vanilla DeiT-S (right). The upper-right time measurements are shown on a quadratic scale to account for the resolution-dependent quadratic cost.
  • Figure 4: Colormap is normalized to [78, 81.7] to highlight differences in the high-accuracy regime; values outside the range saturate.
  • Figure 5: Fraction of layer outputs lying outside the convex hull of input values (CHM) across encoder layers for image, speech, and language models.