Self-Attention Based Semantic Decomposition in Vector Symbolic Architectures

TL;DR

A new variant of the resonator network is introduced, based on self-attention based update rules in the iterative search problem, that enables a larger capacity for associative memory, enabling applications in many tasks like perception based pattern recognition, scene decomposition, and object reasoning.

Abstract

Vector Symbolic Architectures (VSAs) have emerged as a novel framework for enabling interpretable machine learning algorithms equipped with the ability to reason and explain their decision processes. The basic idea is to represent discrete information through high dimensional random vectors. Complex data structures can be built up with operations over vectors such as the "binding" operation involving element-wise vector multiplication, which associates data together. The reverse task of decomposing the associated elements is a combinatorially hard task, with an exponentially large search space. The main algorithm for performing this search is the resonator network, inspired by Hopfield network-based memory search operations. In this work, we introduce a new variant of the resonator network, based on self-attention based update rules in the iterative search problem. This update rule, based on the Hopfield network with log-sum-exp energy function and norm-bounded states, is shown to substantially improve the performance and rate of convergence. As a result, our algorithm enables a larger capacity for associative memory, enabling applications in many tasks like perception based pattern recognition, scene decomposition, and object reasoning. We substantiate our algorithm with a thorough evaluation and comparisons to baselines.

Figures (9)

• Figure 1: Comparison between resonator networks using the old update rule (solid line) and the new attention-based update rule (dotted line). We vary the size of the search space $M$ and plot the mean accuracy over 1000 trials for different numbers of factors $F$ and vector dimensions $D$. We use bipolar vectors for both resonator networks and $\beta=250$ for the attention-based resonator network.
• Figure 2: Comparison between resonator networks using the old update rule (solid line) and the new attention-based update rule (dotted line). We vary the size of the search space $M$ and plot the mean number of iterations before convergence over 1000 trials for different numbers of factors $F$ and vector dimensions $D$. We use bipolar vectors for both resonator networks and set the maximum number of iterations to $0.001M$.
• Figure 3: Comparison of mean accuracy between the original resonator network adapted to FHRR (solid line) and the attention-based resonator network (dotted line).
• Figure 4: Plots of mean accuracy and mean iterations for attention-based resonator networks using bipolar codebooks (solid line) and FHRR codebooks (dotted line) for $F=3$, over 1000 trials.
• Figure 5: Mean accuracy comparison between original and attention-based resonator networks using both bipolar and FHRR hypervectors over multiple configurations of $n$ and $F$, chosen such that $M=n^F\approx 5000$. Averages are computed over 1000 trials. Attention-based resonator networks use $\beta=250$.