Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Holographic Reduced Representations

Calvin Yeung, Zhuowen Zou, Mohsen Imani

TL;DR

The paper addresses the high data and compute costs of deep learning representations by advancing Hyperdimensional Computing through Generalized Holographic Reduced Representations (GHRR). It introduces GHRR as an extension of FHRR with unitary matrix components per dimension, enabling flexible, non-commutative binding and adaptive kernels while preserving core HDC properties. The work provides theoretical guarantees (quasi-orthogonality and kernel-preservation), analyzes binding as holographic projections, and demonstrates through experiments that GHRR improves decoding accuracy for compositional structures and increases memorization capacity relative to FHRR, especially for bound components. The results suggest GHRR offers a data-efficient, interpretable framework for representing and manipulating structured information with controllable non-commutativity and kernel adaptivity, holding promise for scalable neuro-symbolic AI.

Abstract

Deep learning has achieved remarkable success in recent years. Central to its success is its ability to learn representations that preserve task-relevant structure. However, massive energy, compute, and data costs are required to learn general representations. This paper explores Hyperdimensional Computing (HDC), a computationally and data-efficient brain-inspired alternative. HDC acts as a bridge between connectionist and symbolic approaches to artificial intelligence (AI), allowing explicit specification of representational structure as in symbolic approaches while retaining the flexibility of connectionist approaches. However, HDC's simplicity poses challenges for encoding complex compositional structures, especially in its binding operation. To address this, we propose Generalized Holographic Reduced Representations (GHRR), an extension of Fourier Holographic Reduced Representations (FHRR), a specific HDC implementation. GHRR introduces a flexible, non-commutative binding operation, enabling improved encoding of complex data structures while preserving HDC's desirable properties of robustness and transparency. In this work, we introduce the GHRR framework, prove its theoretical properties and its adherence to HDC properties, explore its kernel and binding characteristics, and perform empirical experiments showcasing its flexible non-commutativity, enhanced decoding accuracy for compositional structures, and improved memorization capacity compared to FHRR.

Generalized Holographic Reduced Representations

TL;DR

The paper addresses the high data and compute costs of deep learning representations by advancing Hyperdimensional Computing through Generalized Holographic Reduced Representations (GHRR). It introduces GHRR as an extension of FHRR with unitary matrix components per dimension, enabling flexible, non-commutative binding and adaptive kernels while preserving core HDC properties. The work provides theoretical guarantees (quasi-orthogonality and kernel-preservation), analyzes binding as holographic projections, and demonstrates through experiments that GHRR improves decoding accuracy for compositional structures and increases memorization capacity relative to FHRR, especially for bound components. The results suggest GHRR offers a data-efficient, interpretable framework for representing and manipulating structured information with controllable non-commutativity and kernel adaptivity, holding promise for scalable neuro-symbolic AI.

Abstract

Deep learning has achieved remarkable success in recent years. Central to its success is its ability to learn representations that preserve task-relevant structure. However, massive energy, compute, and data costs are required to learn general representations. This paper explores Hyperdimensional Computing (HDC), a computationally and data-efficient brain-inspired alternative. HDC acts as a bridge between connectionist and symbolic approaches to artificial intelligence (AI), allowing explicit specification of representational structure as in symbolic approaches while retaining the flexibility of connectionist approaches. However, HDC's simplicity poses challenges for encoding complex compositional structures, especially in its binding operation. To address this, we propose Generalized Holographic Reduced Representations (GHRR), an extension of Fourier Holographic Reduced Representations (FHRR), a specific HDC implementation. GHRR introduces a flexible, non-commutative binding operation, enabling improved encoding of complex data structures while preserving HDC's desirable properties of robustness and transparency. In this work, we introduce the GHRR framework, prove its theoretical properties and its adherence to HDC properties, explore its kernel and binding characteristics, and perform empirical experiments showcasing its flexible non-commutativity, enhanced decoding accuracy for compositional structures, and improved memorization capacity compared to FHRR.
Paper Structure (24 sections, 3 theorems, 17 equations, 10 figures)

This paper contains 24 sections, 3 theorems, 17 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Hyperdimensional Computing Basics
  4. Fourier Holographic Reduced Representations
  5. Overview of GHRR
  6. An Implementation of GHRR
  7. Desired Properties
  8. Quasi-orthogonal Representations
  9. Encoding Data
  10. Kernel Properties of GHRR
  11. Binding as holographic projections
  12. Results
  13. A demonstration of non-commutativity
  14. Effect of Q on commutativity
  15. Decoding accuracy for nested structures
  16. ...and 9 more sections

Key Result

Proposition 4.1

Let $\mathbf{A}=\mathbf{Q}_1\mathbf{\Lambda}_1,\mathbf{B}=\mathbf{Q}_2\mathbf{\Lambda}_2\in\mathbb{C}^{m\times m}$ be random matrices where $\mathbf{Q}_j,\mathbf{\Lambda}_j$ are sampled independently of each other for $j=1,2$. Moreover, let $\mathbf{\Lambda}_1=\mathrm{diag}(\lambda_1,...,\lambda_m)$

Figures (10)

  • Figure 1: Comparison between FHRR and GHRR.
  • Figure 2: A histogram of the similarity between randomly sampled hypervectors following the scheme described in Corollary \ref{['cor:orthogonal']}, where $D=1000$, $m=3$, and $p_j=\mathrm{Unif}(0,2\pi)$. Top: Histogram where $\mathbf{Q}_j=\mathbf{Q}_k$ holds for all $j,k=1,...,D$. Middle: histogram where $\mathbf{Q}_j$ for $j=1,...,D$ are randomly sampled. Bottom: histogram of similarity between $H_1$ and $H_1*H_2$.
  • Figure 3: Distribution of $\delta(\phi_1(0),\phi_2(0))$ over encodings $\phi_1,\phi_2$. Top: $\mathbf{Q}$ is fixed across dimensions for $\phi_1,\phi_2$. Bottom: $\mathbf{Q}$ is varied across dimensions for $\phi_1,\phi_2$.
  • Figure 4: A visualization of binding in FHRR and GHRR. Left: FHRR binding as a projection of the diagonal of the outer product matrix. Right: GHRR binding as a projection of the block-diagonal of the outer product matrix.
  • Figure 5: We encode a hypervector $\mathbf{H}=\mathbf{K}_1*(\mathbf{K}_1*\mathbf{V}_1+\mathbf{K}_2*\mathbf{V}_2)+\mathbf{K}_2*(\mathbf{K}_1*\mathbf{V}_3+\mathbf{K}_2*\mathbf{V}_4)$, and retrieve approximate hypervectors $\mathbf{V}_1'=\mathbf{K}_1^{-1}*\mathbf{K}_1^{-1}*\mathbf{H}$, $\mathbf{V}_2'=\mathbf{K}_2^{-1}*\mathbf{K}_1^{-1}*\mathbf{H}$, etc. We then plot the similarities of the retrieved hypervectors against $\phi(x)$. A decoding is successful if there is a peak at the given value and nowhere else. A. A visualization of the encoded structure. B. We use an FHRR encoding, i.e. a commutative encoding. C. We use a GHRR encoding with $m=3$, i.e. a non-commutative encoding.
  • ...and 5 more figures

Theorems & Definitions (5)

  • Proposition 4.1
  • proof
  • Corollary 4.1.1
  • Corollary 4.1.2
  • proof