A Walsh Hadamard Derived Linear Vector Symbolic Architecture

TL;DR

This work introduces the Hadamard-derived linear Binding (HLB), which is designed to have favorable computational efficiency, and efficacy in classic VSA tasks, and perform well in differentiable systems.

Abstract

Vector Symbolic Architectures (VSAs) are one approach to developing Neuro-symbolic AI, where two vectors in $\mathbb{R}^d$ are `bound' together to produce a new vector in the same space. VSAs support the commutativity and associativity of this binding operation, along with an inverse operation, allowing one to construct symbolic-style manipulations over real-valued vectors. Most VSAs were developed before deep learning and automatic differentiation became popular and instead focused on efficacy in hand-designed systems. In this work, we introduce the Hadamard-derived linear Binding (HLB), which is designed to have favorable computational efficiency, and efficacy in classic VSA tasks, and perform well in differentiable systems. Code is available at https://github.com/FutureComputing4AI/Hadamard-derived-Linear-Binding

Figures (6)

• Figure 1: Empirical comparison of the corrected cosine similarity scores between $\phi'_{(+)}$ (on top) and $\phi'_{(-)}$ (on bottom) for varying $n$ and $\rho$ shown in heatmap. The dimension, i.e., $d = 2^n$ is varied from $2$ to $1024$$(n \in \{1, 2, \cdots, 10\})$ and the number of vector pairs bundled is varied from $1$ to $50$. This shows that we can accurately identify when a vector $\boldsymbol{x}$ has been bound to a VSA or not when we keep track of how many pairs of terms $\rho$ are included.
• Figure 2: The area under the accuracy curve due to the change of no. of bundled pairs $\rho$ for dimensions $d$. All the dimensions are chosen to be perfect squares due to the constraint of VTB.
• Figure 3: When repeatedly binding different random (left) or a single vector (right), HLB consistently returns the ideal similarity score of 1 for a present item (top row) and has a constant magnitude (bottom row), avoiding exploding/vanishing values.
• Figure 4: Heatmap of the empirical comparison of the noise components $\eta_i^\circ$ and $\eta_i^\pi$ for varying $n$ and $\rho$ shown in natural logarithm scale. The dimension, i.e., $d = 2^n$ is varied from $2$ to $1024$$(n \in \{1, 2, \cdots, 10\})$ and the number of vector pairs bundled is varied from $2$ to $50$.
• Figure 5: Comparison between the theoretical and experimental relationship of \ref{['thm:norm']}. The norm of the composite representation of the bound vectors is computed for no. of bundled vectors from $1$ to $200$ of dimension $d = 1024$. The figure shows how the experimental value of the norm closely follows the theoretical relation between $\left\Vert \chi_\rho \right\Vert_2$ and $\rho$.

Theorems & Definitions (9)

• Lemma 3.1: Hadamard Properties
• Definition 3.1: Binding and Unbinding
• Theorem 3.1: Inverse Theorem
• proof : Proof of \ref{['thm:inverse']}
• Definition 3.2: Projection
• Theorem 3.2: $\phi$ -- $\rho$ Relationship
• proof : Proof of \ref{['thm:cosine']}
• Theorem B.1: $\chi_\rho$ -- $\rho$ Relationship
• proof : Proof of \ref{['thm:norm']}