Fast and Geometrically Grounded Lorentz Neural Networks

TL;DR

This work tackles the challenge of building fast, geometrically faithful neural networks in hyperbolic space using the Lorentz model. It identifies a fundamental pathology in existing Lorentz linear layers where hyperbolic norms grow only as $O(\ln n)$ with the number of gradient steps, hindering the efficient representation of hierarchies. The authors derive a Lorentz linear layer grounded in the distance-to-hyperplane formulation, combining Lorentzian activations and a caching strategy to achieve linear scaling of hyperbolic distances with training steps and substantially faster inference. They demonstrate both theoretical guarantees (linear-time embedding of depth $h$ trees) and empirical gains (training times of around 2.9x–8.3x faster than prior hyperbolic models, CIFAR-100 training in ~70 minutes) while preserving competitive accuracy on CIFAR-10/100. The results meaningfully close the gap between hyperbolic and Euclidean networks and open avenues for further improvements in normalization and adaptive curvature.

Abstract

Hyperbolic space is quickly gaining traction as a promising geometry for hierarchical and robust representation learning. A core open challenge is the development of a mathematical formulation of hyperbolic neural networks that is both efficient and captures the key properties of hyperbolic space. The Lorentz model of hyperbolic space has been shown to enable both fast forward and backward propagation. However, we prove that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps, nullifying the key advantage of hyperbolic geometry. We propose a new Lorentz linear layer grounded in the well-known ``distance-to-hyperplane" formulation. We prove that our formulation results in the usual linear scaling of output hyperbolic norms with respect to the number of gradient descent steps. Our new formulation, together with further algorithmic efficiencies through Lorentzian activation functions and a new caching strategy results in neural networks fully abiding by hyperbolic geometry while simultaneously bridging the computation gap to Euclidean neural networks. Code available at: https://github.com/robertdvdk/hyperbolic-fully-connected.

Figures (3)

• Figure 1: Number of iterations required to fit a hyperbolic hyperplane with a given hyperbolic distance to the origin.
• Figure 2: Growth of the hyperbolic distance to the origin over the network's layers. Our FGG-LNN grows the distance gradually as depth increases. HCNN has the distance remain relatively constant, until the final BatchNorm layer, where the distance explodes to 9.
• Figure 3: Geometric construction of the Lorentz hyperplanes.(a) A 3D visualization of the Lorentz model $\mathbb{L}^2$ and geodesic hyperplane with discriminative regions in blue and red. The orange vector is the weight parameter $\mathbf{w}$ of the hyperplane. The gray plane $P$ is the set of points in the ambient space Minkowski orthogonal to $\mathbf{v}$. (b) and (c) are planar sections of the hyperboloid in (a) intersecting it with the subspace $\text{span}(\mathbf{e}_1, \mathbf{w})$. This yields a one-dimensional hyperbola. (b) Parametrization of the hyperplane reference point: the weight vector $\mathbf{w}$ and bias $b$ define an anchor point $\mathbf{p}$ on the manifold via the exponential map. The distance from the origin $\mathbf{o}$ to $\mathbf{p}$ is $\frac{b}{\|\mathbf{w}\|_\mathcal{L}}$. (c) The simplified hyperplane condition: the geodesic hyperplane $H_{\mathbf{w},b}$ is the intersection of the hyperboloid with the plane $P$ that are orthogonal to the parallel-transported weight vector v in the ambient Minkowski space $P=\{\mathbf{z} \ | \ \mathbf{z} \circ \mathbf{v} = \mathbf{z} ^T \mathbf{I}_{1,n} \mathbf{v}=0\}$.

Theorems & Definitions (45)

• Proposition 3.1
• proof
• Proposition 3.2
• proof
• Proposition 3.3
• proof
• Proposition 4.1
• proof
• Theorem 4.2
• proof