Information Gravity: A Field-Theoretic Model for Token Selection in Large Language Models

TL;DR

The paper tackles interpretability and reliability challenges in large language models by introducing information gravity, a field-theoretic framework in which a user query carries information mass $M(Q)=\alpha H(Q)+\beta D(Q)+\gamma N(Q)$ that warps the model's semantic space into a gravitational field described by a potential $\Phi(t,Q)=-\log P(t|Q)$. Token generation follows a Boltzmann-like distribution $P(t|Q)\propto\exp(-\Phi(t)/T)$, with the gradient $g(t)=-\nabla\Phi(t)$ guiding the trajectory through a curved semantic landscape; entropy, context depth, and novelty modulate the curvature and, consequently, the likelihood of hallucinations, prompt sensitivity, and temperature-dependent diversity. The framework yields concrete predictions about hallucinations arising from semantic voids, sensitivity to query structure via shifts in the potential landscape, and the role of temperature in balancing exploration and exploitation. The authors propose experimental strategies (semantic landscape visualization, mass-hallucination correlations, temperature variation) and practical applications (adaptive temperature control, information-mass-based quality metrics) that could inform more interpretable and controllable LLM systems.

Abstract

We propose a theoretical model called "information gravity" to describe the text generation process in large language models (LLMs). The model uses physical apparatus from field theory and spacetime geometry to formalize the interaction between user queries and the probability distribution of generated tokens. A query is viewed as an object with "information mass" that curves the semantic space of the model, creating gravitational potential wells that "attract" tokens during generation. This model offers a mechanism to explain several observed phenomena in LLM behavior, including hallucinations (emerging from low-density semantic voids), sensitivity to query formulation (due to semantic field curvature changes), and the influence of sampling temperature on output diversity.

Figures (1)

• Figure 1: Illustration of information gravity in semantic space. A user query creates a gravitational well in the semantic field, attracting token generation toward areas of minimal semantic potential.