Discrete, compositional, and symbolic representations through attractor dynamics

TL;DR

This approach establishes a unified framework that integrates both symbolic and sub-symbolic processing through neural dynamics, a neuro-plausible substrate with proven expressivity in AI, offering a more comprehensive model that mirrors the complex duality of cognitive operations.

Abstract

Symbolic systems are powerful frameworks for modeling cognitive processes as they encapsulate the rules and relationships fundamental to many aspects of human reasoning and behavior. Central to these models are systematicity, compositionality, and productivity, making them invaluable in both cognitive science and artificial intelligence. However, certain limitations remain. For instance, the integration of structured symbolic processes and latent sub-symbolic processes has been implemented at the computational level through fiat methods such as quantization or softmax sampling, which assume, rather than derive, the operations underpinning discretization and symbolicization. In this work, we introduce a novel neural stochastic dynamical systems model that integrates attractor dynamics with symbolic representations to model cognitive processes akin to the probabilistic language of thought (PLoT). Our model segments the continuous representational space into discrete basins, with attractor states corresponding to symbolic sequences, that reflect the semanticity and compositionality characteristic of symbolic systems through unsupervised learning, rather than relying on pre-defined primitives. Moreover, like PLoT, our model learns to sample a diverse distribution of attractor states that reflect the mutual information between the input data and the symbolic encodings. This approach establishes a unified framework that integrates both symbolic and sub-symbolic processing through neural dynamics, a neuro-plausible substrate with proven expressivity in AI, offering a more comprehensive model that mirrors the complex duality of cognitive operations.

Figures (8)

• Figure 1: Model concept. Given an image of the red square as an input $x$, we first sample the maximally informative encoding $z_0 \sim P_\phi(z_0 \mid x)$. Starting at $z_0$ as the initial condition, we iteratively sample subsequent states according to the forward policy $z_{t+1} \sim P_\theta(z_{t+1} \mid z_t)$. The stochasticity allows the model to sample a diverse set of trajectories for a given input, while the attractor dynamics pull the trajectories towards sentence embeddings $\hat{z}_s$, so that the distribution of terminal states $P_\theta(z_T \mid x)$ (represented as the density in the figure) forms discrete attractor basins around the embeddings. Each sentence embedding $\hat{z}_s$ corresponds to a token sequence $s$, and the relative density around $\hat{z}_s$ corresponds to how well the sentence describes the red square image. In this example, the states around the embedding for the token sequence that translates to 'top right large red square' is sampled most frequently, while states around the embedding representing 'heart' are effectively never sampled. Other inputs (shown on the right) map to their own respective encodings $z_0$ and produce trajectories towards appropriate sentence embeddings, e.g., 'heart' for the images containing hearts. Note that we show English words in this diagram for interpretability; the model uses arbitrary tokens with emergent semantics without exposure to any formal language.
• Figure 2: Projections of sample trajectories to two principal dimensions for the HBV dataset. Black dots indicate initial conditions $z_0$. Blue dots indicate terminal states $z_T$. Red dots indicate sentence embeddings $\hat{z}_w$. (a) Trajectories sampled using 5 exemplars from 10 different classes. (b) Trajectories sampled from a single exemplar. Note that the two figures are scaled differently.
• Figure 3: (A) Simplified HBV dataset with 16-bit vectors and a depth-3 tree. (A1) The underlying hierarchical structure. Nodes in red indicate held-out classes. (A2) Prototype vectors and sample exemplars. White and black bits indicate where the prototype and sample are both 1 and 0 respectively. Light gray bits indicate where the sample is 1 but the prototype is 0. Dark gray bits indicate where the sample is 0 but the prototype is 1. (A3) Feature correlation matrix. (B) Samples from the dSprites dataset. Left three columns show samples from the training set and the right three columns from the held-out sets.
• Figure 4: Distance between a sentence embedding $\hat{z}$ and the post-dynamics state $z'_{T'}$ after adding varying degrees of perturbation. 'Original $\hat{z}_s$' panels show distances to the original sentence embedding, while 'Nearest $\hat{z}_n$' panels reflect distances to the closest sentence embedding post-dynamics. Thin lines represent distance percentiles and red dashed lines mark the median distances, with the inflection points suggesting the attractor basin boundaries. Higher percentile lines at small perturbations reflect trajectories that leave the attractor basin due to stochastic dynamics.
• Figure 5: Correlation of predicted features between original and nearest post-trajectory sentence embeddings $\hat{z}_s$ and $\hat{z}_n$. Each panel indicates a different image feature--R, G, B for color channels; shape; scale; X and Y for position--with the Pearson correlation coefficient (r). The dashed line represents the line of best fit.