On the Role of Noise in Factorizers for Disentangling Distributed Representations

TL;DR

This work investigates how noise can enhance factorization of distributed representations in vector-symbolic architectures. It introduces two variants—In-Memory Factorizer (IMF) that exploits analog memory noise during iterations, and Asymmetric Codebook Factorizer (ACF) that initializes one codebook with noisy perturbations—demonstrating substantial improvements in operational capacity over baseline resonator networks. Across $F \in \{2,3,4\}$ and increasing search spaces, IMF and ACF achieve at least a 50x expansion in feasible problem size, with tradeoffs in energy, latency, and hardware complexity. The findings suggest practical, accelerator-friendly paths to scalable factorization in large-scale, noisy computing environments, including digital implementations via initialization noise and analog implementations via intrinsic memory noise. The work also outlines convergence detection and thresholding strategies that support efficient decoding and potential synergy between IMF and ACF in future designs.

Abstract

To efficiently factorize high-dimensional distributed representations to the constituent atomic vectors, one can exploit the compute-in-superposition capabilities of vector-symbolic architectures (VSA). Such factorizers however suffer from the phenomenon of limit cycles. Applying noise during the iterative decoding is one mechanism to address this issue. In this paper, we explore ways to further relax the noise requirement by applying noise only at the time of VSA's reconstruction codebook initialization. While the need for noise during iterations proves analog in-memory computing systems to be a natural choice as an implementation media, the adequacy of initialization noise allows digital hardware to remain equally indispensable. This broadens the implementation possibilities of factorizers. Our study finds that while the best performance shifts from initialization noise to iterative noise as the number of factors increases from 2 to 4, both extend the operational capacity by at least 50 times compared to the baseline factorizer resonator networks. Our code is available at: https://github.com/IBM/in-memory-factorizer

Figures (3)

• Figure 1: Factorization of perceptual representations: color position and object type using factorizers. Taking the product vector as input, and starting from initial estimate vectors the factorizer undergoes unbinding (UB), associative search (AS), attention activation (AA), and reconstruction (RC) phases to iteratively refine the estimate. AS and RC take the biggest share of the computing and memory. They map to a predominantly MVM operation.
• Figure 2: Implementing noise during a decoding iteration of a single factor of factorizer. (a) The codebooks are implemented on an analog memory device crossbar array which introduce intrinsic noise to each iteration of both the AS and RC phases. (b) The codebooks are implemented on digital memory devices. The second codebook used in the RC phase is made asymmetric from the first codebook in the AS phase by a bit flip mask perturbation. (c) The resulting attention using analog IMC. It can have a continuous distribution. (d) The resulting attention using asymmetric codebooks. It follows a discrete distribution.
• Figure 3: The number of iterations (top row) and accuracy (bottom row) for three variants of the factorizer: baseline resonator (in blue) FradyResonator2020, in-memory factorizer (in orange) langenegger2023imf, and our asymmetric codebook factorizer (in green). The left, center, and right column results correspond to 2,3, and 4-factor scenarios, respectively. The regions that meet operational capacity criteria (i.e. $\geq$99% accuracy) are plotted with solid lines while other regions are plotted with dotted lines.