Developing a Foundation of Vector Symbolic Architectures Using Category Theory
Nolan P Shaw, P Michael Furlong, Britt Anderson, Jeff Orchard
TL;DR
This paper addresses the lack of a solid mathematical foundation for Vector Symbolic Architectures (VSAs) by introducing a category-theoretic framework that generalises vectors to co-presheaves and derives VSA operations via right Kan extensions of external tensor products. The key idea is to decouple indices from values, encode bind and bundle as $v ensor w = ext{Ran}_{ar{ ensor}}(v ar{ ensor} w)$ and $v oxplus w = ext{Ran}_{ar{ ensor}}(v ar{oxplus} w)$, and to express similarity through a dagger-structured, enriched value category, culminating in an element-wise, information-preserving operational core. Worked examples show that this formalisation recovers standard VSAs (e.g., tensor product representations) and accommodates finite-index scenarios, while suggesting new VSA designs guided by Kan-extensions. The work lays a principled foundation for future category-theoretic analyses of VSAs and their compositional capabilities, with potential connections to Hilbert-space formalisms and more rigorous treatment of analogy and structure-building in symbolic cognition.
Abstract
Connectionist approaches to machine learning, \emph{i.e.} neural networks, are enjoying a considerable vogue right now. However, these methods require large volumes of data and produce models that are uninterpretable to humans. An alternative framework that is compatible with neural networks and gradient-based learning, but explicitly models compositionality, is Vector Symbolic Architectures (VSAs). VSAs are a family of algebras on high-dimensional vector representations. They arose in cognitive science from the need to unify neural processing and the kind of symbolic reasoning that humans perform. While machine learning methods have benefited from category-theoretical analyses, VSAs have not yet received similar treatment. In this paper, we present a first attempt at applying category theory to VSAs. Specifically, We generalise from vectors to co-presheaves, and describe VSA operations as the right Kan extensions of the external tensor product. This formalisation involves a proof that the right Kan extension in such cases can be expressed as simple, element-wise operations. We validate our formalisation with worked examples that connect to current VSA implementations, while suggesting new possible designs for VSAs.