Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing

Mohamed Mejri, Chandramouli Amarnath, Abhijit Chatterjee

TL;DR

This work proposes RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling and binding allowing both object-level features and relational representations to coexist within the same structure without interfering with one another.

Abstract

Modern transformer-based encoder-decoder architectures struggle with reasoning tasks due to their inability to effectively extract relational information between input objects (data/tokens). Recent work introduced the Abstractor module, embedded between transformer layers, to address this gap. However, the Abstractor layer while excelling at capturing relational information (pure relational reasoning), faces challenges in tasks that require both object and relational-level reasoning (partial relational reasoning). To address this, we propose RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling (summation) and binding (Hadamard product) allowing both object-level features and relational representations to coexist within the same structure without interfering with one another. RESOLVE is driven by a novel attention mechanism that operates in a bipolar high dimensional space, allowing fast attention score computation compared to the state-of-the-art. By leveraging this design, the model achieves both low compute latency and memory efficiency. RESOLVE also offers better generalizability while achieving higher accuracy in purely relational reasoning tasks such as sorting as well as partial relational reasoning tasks such as math problem-solving compared to state-of-the-art methods.

RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing

TL;DR

This work proposes RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling and binding allowing both object-level features and relational representations to coexist within the same structure without interfering with one another.

Abstract

Modern transformer-based encoder-decoder architectures struggle with reasoning tasks due to their inability to effectively extract relational information between input objects (data/tokens). Recent work introduced the Abstractor module, embedded between transformer layers, to address this gap. However, the Abstractor layer while excelling at capturing relational information (pure relational reasoning), faces challenges in tasks that require both object and relational-level reasoning (partial relational reasoning). To address this, we propose RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling (summation) and binding (Hadamard product) allowing both object-level features and relational representations to coexist within the same structure without interfering with one another. RESOLVE is driven by a novel attention mechanism that operates in a bipolar high dimensional space, allowing fast attention score computation compared to the state-of-the-art. By leveraging this design, the model achieves both low compute latency and memory efficiency. RESOLVE also offers better generalizability while achieving higher accuracy in purely relational reasoning tasks such as sorting as well as partial relational reasoning tasks such as math problem-solving compared to state-of-the-art methods.

Paper Structure

This paper contains 42 sections, 1 equation, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Overview
  4. Relational Bottleneck and VSA approach Modeling
  5. RESOLVE: HD-Encoder and HD-Attention Mechanism With Hypervector Bundling
  6. RESOLVE: Architecture Overview
  7. Experiments
  8. Single Output Purely Relational Tasks
  9. Order relations: modeling asymmetric relations
  10. SET: modeling multi-dimensional relations with pre-processed objects
  11. Single Output Partially Relational Tasks
  12. SET: modeling multi-dimensional relations with low-processed objects
  13. MNIST-MATH: extracting mathematical rules from a pair of digit images
  14. Object-sorting: Purely Relational Sequence-to-Sequence Tasks
  15. Math problem-solving: partially-relational sequence-to-sequence tasks
  16. ...and 27 more sections

Figures (13)

  • Figure 1: Example of purely relational task: Pairwise Ordering
  • Figure 2: Two examples of partially relational tasks(Figure \ref{['partial_1']} and \ref{['partial_2']})
  • Figure 3: Comparison of a relational bottleneck approach applied on the transformer (Figure \ref{['abstractor_O']}) separating object-level features while keeping only abstract features with a vector symbolic architecture alternative to the relational bottleneck using binding to mix object and abstract level information in high dimensional (HD) space with low interference (Figure \ref{['ABOB_O']})
  • Figure 4: Comparison between SelfAttentionvaswani2017attention\ref{['fig:Self_attention']}, RelationalCrossAttentionabstractor\ref{['fig:Relational_cross_attention']} and our VSA approach \ref{['fig:VSA']}. We show a single head of multi-attention for brevity. The object-related operations are in red and the relational-related (abstract/symbolic) operations are in turquoise.
  • Figure 5: HD-Encoder $\phi_{\texttt{HD}}$ and HD-Attention($\texttt{O}_{\texttt{1\dots N}}$)
  • ...and 8 more figures