Discrete Semantic States and Hamiltonian Dynamics in LLM Embedding Spaces

TL;DR

This work investigates whether LLM embedding spaces exhibit discrete semantic states by importing a Hamiltonian formalism to analyze semantic transformations under $L_2$-normalization. It derives how cosine similarity $S_C$ relates to perturbations and explores direct and indirect transitions through Hamiltonians, including a rank-1 $H'$ with diagonalization leading to a time-evolving single dominant mode. The authors validate their framework numerically on a 768-dimensional embedding model and discuss quantum-inspired concepts such as a zero-point energy analogue and KvN mechanics as lenses for understanding semantic dynamics. They argue that this mathematical perspective could yield new methods to mitigate hallucinations and inform efficient, reliable NLP via symmetry-enforcing and regularization approaches, while acknowledging the exploratory nature of the analogies.

Abstract

We investigate the structure of Large Language Model (LLM) embedding spaces using mathematical concepts, particularly linear algebra and the Hamiltonian formalism, drawing inspiration from analogies with quantum mechanical systems. Motivated by the observation that LLM embeddings exhibit distinct states, suggesting discrete semantic representations, we explore the application of these mathematical tools to analyze semantic relationships. We demonstrate that the L2 normalization constraint, a characteristic of many LLM architectures, results in a structured embedding space suitable for analysis using a Hamiltonian formalism. We derive relationships between cosine similarity and perturbations of embedding vectors, and explore direct and indirect semantic transitions. Furthermore, we explore a quantum-inspired perspective, deriving an analogue of zero-point energy and discussing potential connections to Koopman-von Neumann mechanics. While the interpretation warrants careful consideration, our results suggest that this approach offers a promising avenue for gaining deeper insights into LLMs and potentially informing new methods for mitigating hallucinations.

Figures (5)

• Figure 1: Visualization of a 16-dimensional embedding vector, representing a semantic concept within an LLM's embedding space. The x-axis indicates the dimension index, while the y-axis represents the value of each dimension, scaled between -1 and 1. Although the specific values are arbitrary for illustrative purposes, the vector is L2-normalized, meaning its magnitude is 1. The horizontal lines depict the magnitude of each dimension, illustrating the discrete nature of the embedding vector components. This provides a visual representation of the structured nature of the embedding space.
• Figure 2: Visualization of a 16-dimensional embedding vector representing a semantic concept. Horizontal lines show dimension magnitudes (scaled -1 to 1). A red line indicates the first perturbed state, where the value of one dimension is inverted compared to the fully dissimilar state, illustrating the smallest possible semantic change.
• Figure 3: IIlustration of the embedding vector $\mathbf{b}$ after a specific perturbation. Horizontal lines represent the values of each dimension in the vector, scaled from -1 to 1. Red lines highlight dimensions $j$ and $i$, which have undergone similar perturbations, effectively swapping their original values. This type of perturbation corresponds to a transition to a higher excited state in the LLM embedding space.
• Figure 4: Quantum-inspired representation of semantic relationships in LLM embedding space. Three semantic states, $\lvert1\rangle$, $\lvert2\rangle$, and $\lvert3\rangle$, are visualized as energy levels with associated population distributions, drawing an analogy to quantum mechanical systems. The arrows represent effective Hamiltonians governing transitions between these states, with $H_{1 \rightarrow 3}$ indicating a direct transformation from $\lvert1\rangle$ to $\lvert3\rangle$, and $H_{1 \rightarrow 2}$ and $H_{2 \rightarrow 3}$ representing sequential transformations through the intermediate state $\lvert2\rangle$, suggesting a more complex semantic relationship.
• Figure 5: Visualization of the time evolution of the quantum state $\lvert\psi(t)\rangle$. The x and y axes represent the real and imaginary components of the time-evolving term $A_1 e^{-i t / \hbar} \lvert1\rangle$, tracing a circle in the complex plane (red dashed line). The z-axis represents a simplified sum of static components $\sum_{n=2}^N A_n \lvert n\rangle$. The blue line shows the trajectory of the superposition $\lvert\psi(t)\rangle$. The green sphere indicates the ground state, consisting only of static components.