The Geometry of Reasoning: Flowing Logics in Representation Space
Yufa Zhou, Yixiao Wang, Xunjian Yin, Shuyan Zhou, Anru R. Zhang
TL;DR
The paper introduces a differential-geometric framework for understanding LLM reasoning as smooth flows in the embedding space $\mathcal{R}$, with logic acting as a local controller of flow velocity. By separating logical structure from semantic carriers via a concept space $\mathcal{C}$ and a canonical alignment between semantic and representation trajectories, it provides formal definitions and tools to analyze reasoning dynamics. Empirical results on a controlled dataset show that velocity and curvature invariants align with the same logical skeleton across topics and languages, validating the claim that logic governs reasoning flows beyond surface semantics. The work offers a conceptual foundation and practical tools for interpretability, steering, and analysis of LLM reasoning, with implications for retrieval, alignment, and architecture design that parameterize latent flows.
Abstract
We study how large language models (LLMs) ``think'' through their representation space. We propose a novel geometric framework that models an LLM's reasoning as flows -- embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomenon, offering a new lens for interpretability and formal analysis of LLMs' behavior.