Hyperbolic Large Language Models

TL;DR

This survey consolidates the burgeoning field of Hyperbolic LLMs (HypLLMs), arguing that negative-curvature geometry better captures hierarchical and tree-like structures common in language, graphs, and multimodal data. It introduces a four-category taxonomy (exp/log-based hybrids, hyperbolic fine-tuned models, fully hyperbolic LLMs, hyperbolic state-space models) and surveys mathematical foundations, optimization, benchmarks, and emerging applications. The work compiles representative models, performance trends on math and hierarchical reasoning tasks, and discusses practical challenges such as numerical stability, computational overhead, and hardware support, while proposing directions like mixture-of-curvature and unified benchmarks. It also highlights cross-domain applications in vision, multimodal learning, neuroscience, and biomedicine, illustrating hyperbolic geometry as a unifying paradigm for scalable hierarchical representation learning.

Abstract

Large language models (LLMs) have achieved remarkable success and demonstrated superior performance across various tasks, including natural language processing (NLP), weather forecasting, biological protein folding, text generation, and solving mathematical problems. However, many real-world data exhibit highly non-Euclidean latent hierarchical anatomy, such as protein networks, transportation networks, financial networks, brain networks, and linguistic structures or syntactic trees in natural languages. Effectively learning intrinsic semantic entailment and hierarchical relationships from these raw, unstructured input data using LLMs remains an underexplored area. Due to its effectiveness in modeling tree-like hierarchical structures, hyperbolic geometry -- a non-Euclidean space -- has rapidly gained popularity as an expressive latent representation space for complex data modeling across domains such as graphs, images, languages, and multi-modal data. Here, we provide a comprehensive and contextual exposition of recent advancements in LLMs that leverage hyperbolic geometry as a representation space to enhance semantic representation learning and multi-scale reasoning. Specifically, the paper presents a taxonomy of the principal techniques of Hyperbolic LLMs (HypLLMs) in terms of four main categories: (1) hyperbolic LLMs through exp/log maps; (2) hyperbolic fine-tuned models; (3) fully hyperbolic LLMs, and (4) hyperbolic state-space models. We also explore crucial potential applications and outline future research directions. A repository of key papers, models, datasets, and code implementations is available at https://github.com/sarangp2402/Hyperbolic-LLM-Models.

Figures (7)

• Figure 1: Three commonly used representation spaces: Euclidean space, where traditional LLMs operate; spherical geometry, which constrains embeddings to bounded surfaces; and hyperbolic space, the basis of Hyperbolic LLMs, which naturally models hierarchical structures.
• Figure 1: Hybrid models use exponential/logarithmic mappings between Euclidean and hyperbolic spaces with options for Poincaré ball or Lorentz embeddings.
• Figure 1: Applications of hyperbolic large language models (HypLLMs) across different domains. Each bubble corresponds to a domain, with included HypLLMs and their corresponding authors.
• Figure 2: Illustration of the two primary hyperbolic representation models used in HypLLMs: the Lorentz model (top left) and the Poincaré ball model (right). The Lorentz model represents hyperbolic space on a single sheet of hyperboloid embedded in Minkowski space $\mathcal{L}_r^{n+1}$, offering closed-form geodesic computations and improved numerical stability. The lower oval shows Poincaré ball disk projection in space $\mathcal{B}_r^n$, this a conformal model where geodesics are circular arcs and points near the boundary encode fine-grained leaf-level hierarchy. These geometric representations provide the foundation for hierarchical embeddings in hyperbolic LLMs.
• Figure 2: Fine-tuned approaches adapt frozen pre-trained models with hyperbolic adapters using curvature-constrained updates.