Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LARS-VSA: A Vector Symbolic Architecture For Learning with Abstract Rules

Mohamed Mejri, Chandramouli Amarnath, Abhijit Chatterjee

TL;DR

LARS-VSA introduces a vector symbolic architecture leveraging hyperdimensional computing to address the relational bottleneck in abstract reasoning. It develops an HD attention mechanism (HDSymbolicAttention) with explicit vector binding between object representations and symbolic rules, including a binarized variant for efficiency, and a architecture that aggregates multi-head HD relations with minimal information loss. Across discriminative relational tasks, purely relational sequence-to-sequence tasks, and math problem-solving, LARS-VSA achieves higher accuracy and substantially lower memory and latency than baselines like Abstractor and Transformer, especially in data-scarce regimes. The approach demonstrates robust relational reasoning with low overhead, offering a scalable neuro-symbolic alternative for learning abstract rules, though it notes temporal interference and future work to push toward fully HD-only decoding.

Abstract

Human cognition excels at symbolic reasoning, deducing abstract rules from limited samples. This has been explained using symbolic and connectionist approaches, inspiring the development of a neuro-symbolic architecture that combines both paradigms. In parallel, recent studies have proposed the use of a "relational bottleneck" that separates object-level features from abstract rules, allowing learning from limited amounts of data . While powerful, it is vulnerable to the curse of compositionality meaning that object representations with similar features tend to interfere with each other. In this paper, we leverage hyperdimensional computing, which is inherently robust to such interference to build a compositional architecture. We adapt the "relational bottleneck" strategy to a high-dimensional space, incorporating explicit vector binding operations between symbols and relational representations. Additionally, we design a novel high-dimensional attention mechanism that leverages this relational representation. Our system benefits from the low overhead of operations in hyperdimensional space, making it significantly more efficient than the state of the art when evaluated on a variety of test datasets, while maintaining higher or equal accuracy.

LARS-VSA: A Vector Symbolic Architecture For Learning with Abstract Rules

TL;DR

LARS-VSA introduces a vector symbolic architecture leveraging hyperdimensional computing to address the relational bottleneck in abstract reasoning. It develops an HD attention mechanism (HDSymbolicAttention) with explicit vector binding between object representations and symbolic rules, including a binarized variant for efficiency, and a architecture that aggregates multi-head HD relations with minimal information loss. Across discriminative relational tasks, purely relational sequence-to-sequence tasks, and math problem-solving, LARS-VSA achieves higher accuracy and substantially lower memory and latency than baselines like Abstractor and Transformer, especially in data-scarce regimes. The approach demonstrates robust relational reasoning with low overhead, offering a scalable neuro-symbolic alternative for learning abstract rules, though it notes temporal interference and future work to push toward fully HD-only decoding.

Abstract

Human cognition excels at symbolic reasoning, deducing abstract rules from limited samples. This has been explained using symbolic and connectionist approaches, inspiring the development of a neuro-symbolic architecture that combines both paradigms. In parallel, recent studies have proposed the use of a "relational bottleneck" that separates object-level features from abstract rules, allowing learning from limited amounts of data . While powerful, it is vulnerable to the curse of compositionality meaning that object representations with similar features tend to interfere with each other. In this paper, we leverage hyperdimensional computing, which is inherently robust to such interference to build a compositional architecture. We adapt the "relational bottleneck" strategy to a high-dimensional space, incorporating explicit vector binding operations between symbols and relational representations. Additionally, we design a novel high-dimensional attention mechanism that leverages this relational representation. Our system benefits from the low overhead of operations in hyperdimensional space, making it significantly more efficient than the state of the art when evaluated on a variety of test datasets, while maintaining higher or equal accuracy.
Paper Structure (42 sections, 1 theorem, 6 equations, 16 figures, 1 algorithm)

This paper contains 42 sections, 1 theorem, 6 equations, 16 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Related Work: Self Attention and Relational Cross Attention
  4. LARS-VSA: Attention Mechanism With Hypervector Bundling
  5. LARS-VSA: Symbolic Reasoning with Hypervectors
  6. Binarized HD attention score
  7. LARS-VSA Architecture
  8. Experiments
  9. Discriminative Relational Tasks
  10. Order relations: modeling asymmetric relations.
  11. SET: modeling multi-dimensional relations.
  12. ABA: Modeling Identity Relations
  13. Object-sorting: Purely Relational Sequence-to-Sequence Tasks
  14. Math problem-solving: partially-relational sequence-to-sequence tasks
  15. Conclusion, Limitation and Future Directions
  16. ...and 27 more sections

Key Result

Lemma 1

Given two D-dimensional vectors $h_1$ and $h_2$ that are bipolar meaning ${h_{1/2}}_i \in \{ -1,+1\} \; \; \forall i \in \llbracket 1, D \rrbracket$ and let $B x: \mapsto \frac{x+1}{2}$ maps bipolar words to binary domain, $\mathrm{\&}$ is the binary AND and $\left\lVert.\right\rVert_0$ is the zero-

Figures (16)

  • Figure 1: Relational cross-attention and self-attention. We show a single head of multi-attention. In blue, the objects related by $r(.,.)$; in green, the inductive bias denoted by symbols
  • Figure 2: $\mathrm{HDSymbolicAttention(O)}$$\otimes$$h_S$
  • Figure 3: $\mathrm{HDSymbolicAttention}(O_{1..N})$
  • Figure 4: Binarized Attention Score Function $r_{12} = cos(h_{{O}_1},h_{{O}_1}\oplus h_{{O}_2})$
  • Figure 5: Examples of evaluated LARS-VSA based pipelines
  • ...and 11 more figures

Theorems & Definitions (2)

  • Lemma 1
  • proof