Semantic Wave Functions: Exploring Meaning in Large Language Models Through Quantum Formalism

TL;DR

This paper explores the analogy between LLM embedding spaces and quantum mechanics, positing that LLMs operate within a quantized semantic space where words and phrases behave as quantum states, and introduces a ”semantic wave function” to formalize this quantum-derived representation.

Abstract

Large Language Models (LLMs) encode semantic relationships in high-dimensional vector embeddings. This paper explores the analogy between LLM embedding spaces and quantum mechanics, positing that LLMs operate within a quantized semantic space where words and phrases behave as quantum states. To capture nuanced semantic interference effects, we extend the standard real-valued embedding space to the complex domain, drawing parallels to the double-slit experiment. We introduce a "semantic wave function" to formalize this quantum-derived representation and utilize potential landscapes, such as the double-well potential, to model semantic ambiguity. Furthermore, we propose a complex-valued similarity measure that incorporates both magnitude and phase information, enabling a more sensitive comparison of semantic representations. We develop a path integral formalism, based on a nonlinear Schrödinger equation with a gauge field and Mexican hat potential, to model the dynamic evolution of LLM behavior. This interdisciplinary approach offers a new theoretical framework for understanding and potentially manipulating LLMs, with the goal of advancing both artificial and natural language understanding.

Figures (4)

• Figure 1: Principal Component Analysis (PCA) projection of word embeddings for "dogs," "cats," and example prompts. This visualization demonstrates the limitations of real-valued embeddings and cosine similarity in capturing nuanced semantic relationships. While prompts related to "cats" generally cluster closer to the "cats" embedding, the overlap between clusters suggests that context and subtle differences in meaning are not fully captured by this approach.
• Figure 2: The double-slit experiment, illustrating the importance of phase information in capturing interference effects. In analogy to LLMs, this demonstrates why complex-valued representations are necessary to model the nuanced semantic relationships that cannot be captured by real-valued embeddings alone.
• Figure 3: The Mexican hat potential, illustrating the concept of spontaneous symmetry breaking. In the context of LLMs, this potential can be used to model the emergence of stable semantic meanings, where the system "chooses" a particular interpretation despite the potential's inherent symmetry.
• Figure 4: The double-well potential, illustrating how context can influence the interpretation of a word in LLMs. The two minima represent different meanings, and the "tunneling" effect represents the LLM's ability to switch between these meanings based on the surrounding context.