Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text Has Curvature

Karish Grover, Hanqing Zeng, Yinglong Xia, Christos Faloutsos, Geoffrey J. Gordon

TL;DR

This work argues that text does indeed have curvature, and shows how to detect it, define it, and use it, and establishes a text-native curvature paradigm, making curvature measurable and practically useful.

Abstract

Does text have an intrinsic curvature? Language is increasingly modeled in curved geometries - hyperbolic spaces for hierarchy, mixed-curvature manifolds for compositional structure - yet a basic scientific question remains unresolved: what does curvature mean for text itself, in a way that is native to language rather than an artifact of the embedding space we choose? We argue that text does indeed have curvature, and show how to detect it, define it, and use it. To this end, we propose Texture, a text-native, word-level discrete curvature signal, and make three contributions. (a) Existence: We provide empirical and theoretical certificates that semantic inference in natural corpora is non-flat, i.e. language has inherent curvature. (b) Definition: We define Texture by reconciling left- and right-context beliefs around a masked word through a Schrodinger bridge, yielding a curvature field that is positive where context focuses meaning and negative where it fans out into competing continuations. (c) Utility: Texture is actionable: it serves as a general-purpose measurement and control primitive enabling geometry without geometric training; we instantiate it on two representative tasks, improving long-context inference through curvature-guided compression and retrieval-augmented generation through curvature-guided routing. Together, our results establish a text-native curvature paradigm, making curvature measurable and practically useful.

Text Has Curvature

TL;DR

This work argues that text does indeed have curvature, and shows how to detect it, define it, and use it, and establishes a text-native curvature paradigm, making curvature measurable and practically useful.

Abstract

Does text have an intrinsic curvature? Language is increasingly modeled in curved geometries - hyperbolic spaces for hierarchy, mixed-curvature manifolds for compositional structure - yet a basic scientific question remains unresolved: what does curvature mean for text itself, in a way that is native to language rather than an artifact of the embedding space we choose? We argue that text does indeed have curvature, and show how to detect it, define it, and use it. To this end, we propose Texture, a text-native, word-level discrete curvature signal, and make three contributions. (a) Existence: We provide empirical and theoretical certificates that semantic inference in natural corpora is non-flat, i.e. language has inherent curvature. (b) Definition: We define Texture by reconciling left- and right-context beliefs around a masked word through a Schrodinger bridge, yielding a curvature field that is positive where context focuses meaning and negative where it fans out into competing continuations. (c) Utility: Texture is actionable: it serves as a general-purpose measurement and control primitive enabling geometry without geometric training; we instantiate it on two representative tasks, improving long-context inference through curvature-guided compression and retrieval-augmented generation through curvature-guided routing. Together, our results establish a text-native curvature paradigm, making curvature measurable and practically useful.
Paper Structure (112 sections, 9 theorems, 88 equations, 4 figures, 7 tables, 3 algorithms)

This paper contains 112 sections, 9 theorems, 88 equations, 4 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. The gap: no text-native curvature formalism.
  3. Our approach: curvature as a property of two-sided inference.
  4. Contributions.
  5. Related Work
  6. Curved representations for language and foundation models.
  7. Discrete curvature on graphs.
  8. Entropic transport and Schrödinger bridges as a reconciliation primitive.
  9. Long-context efficiency: compression, retrieval, and adaptive routing.
  10. Preliminaries
  11. Text, slots, and context radii.
  12. Finite slot state spaces, tail bucket, and belief extractors.
  13. KL divergence.
  14. Markov kernels, stationarity, and reversibility.
  15. Schrödinger bridges and Sinkhorn scaling on finite supports.
  16. ...and 97 more sections

Key Result

Theorem 4.1

Assume $\mu_i(L,R)\in\Delta^\circ(\mathcal{S}_i)$ on a rectangular grid domain. Then $\Omega_{i,s}(L,R)\equiv 0$ for all $s$ and all unit squares if and only if there exist functions $\alpha_{i,s}(\cdot)$ and $\beta_{i,s}(\cdot)$ such that $u_{i,s}(L,R)=u_{i,s}(0,0)+\alpha_{i,s}(L)+\beta_{i,s}(R).$

Figures (4)

  • Figure 1: Texture overview.(a) Our primitive object is a two-sided slot with prefix/suffix beliefs. (b) Natural text exhibits non-flat two-sided belief composition (order-sensitivity and evidence interaction), motivating curvature and enabling definition-independent existence tests (Section \ref{['sec:existence']}). (c) Texture defines a text-native signed curvature by comparing endpoints to a conservative symmetric reconciliation (Section \ref{['sec:texture']}). (d) The resulting curvature field is a general-purpose control signal: we instantiate it on pruning and retrieval, but it applies broadly to language tasks requiring context-aware resource allocation (Section \ref{['sec:utility']}).
  • Figure 2: Empirical falsification of flatness nulls (Panels A--B). Distributions are over $1000$ slots per corpus (WikiText-2 and OpenWebText) using distilroberta-base. (A) Holonomy magnitude $h_i$ is consistently higher on natural text than on suffix-swap and local-shuffle controls, indicating order-sensitive evidence updates in two-sided inference. (B) CEI shows analogous separation, indicating non-additive evidence composition relative to the PoE null. Full diagnostics, medians/effect sizes, and bootstrap CIs appear in Appendix \ref{['app:existence-empirical']}.
  • Figure 3: Texture operator pipeline. Given a word-in-context slot, we (a) extract boundary beliefs $\mu_i^L,\mu_i^R$ on $\mathcal{S}_i$, (b) build a neutral kernel $K_i$ from semantic costs, (c) compute the Schrödinger-bridge midpoint $\mu_i^{\mathrm{mid}}$ as conservative two-sided reconciliation, and (d) read off signed curvature from the midpoint's behavior (focus vs fan-out).
  • Figure 4: Existence diagnostics (appendix: Panels C--D).(C) Holonomy sanity check (discrete telescoping / "discrete Stokes"): rectangle holonomy equals the sum of unit-square holonomies over the rectangle (Appendix Theorem \ref{['thm:holonomy-flatness']}). Points concentrate near the identity line, validating the loop computation. (D) Scale-sensitive motivation plot: mean interaction signal versus token frequency rank (log--log, binned), comparing real text to coherence-destroying controls. This plot is not used as a certificate, but provides an inference-native analogue of distributional motivations for curved language structure.

Theorems & Definitions (16)

  • Theorem 4.1: Holonomy flatness null; see Theorem \ref{['thm:holonomy-flatness']}
  • Theorem 4.2: PoE flatness null; see Theorem \ref{['thm:poe-certificate']}
  • Theorem 3.1: Holonomy characterization of log-odds separability
  • proof
  • Theorem 3.2: Product-of-Experts characterization of evidence additivity
  • proof
  • Lemma 4.1: Reversibility of the Gibbs row-normalized kernel
  • proof
  • Theorem 4.2: Two-step bridge structure and computability
  • proof
  • ...and 6 more