Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Space

Max van Spengler, Pascal Mettes

TL;DR

Addressing the need for faithful, GPU-friendly hyperbolic embeddings of tree-structured data, the paper proposes HS-DTE, a highly separated Delaunay-based constructive embedding that optimizes hyperspherical separation, and HypFPE, a floating point expansion framework for high-precision computations on GPUs. HS-DTE uses a minimal angle maximization objective to place children on hyperspheres with improved separation, enabling lower distortion across dimensions; HypFPE provides ulp-nonoverlapping expansions and new routines to compute hyperbolic distances via acosh and atanh with enhanced precision. Empirically, HS-DTE outperforms existing constructive and optimization-based methods on complete m-ary and phylogenetic trees, and HypFPE further boosts accuracy when combined with HS-DTE or other constructive methods, while preserving GPU compatibility. The work also delivers theoretical error guarantees for FPE-based distance computations and releases software libraries for arbitrary-dimensional hyperbolic embeddings and GPU-optimized precision arithmetic, broadening the practical impact for hyperbolic neural networks.

Abstract

Embedding tree-like data, from hierarchies to ontologies and taxonomies, forms a well-studied problem for representing knowledge across many domains. Hyperbolic geometry provides a natural solution for embedding trees, with vastly superior performance over Euclidean embeddings. Recent literature has shown that hyperbolic tree embeddings can even be placed on top of neural networks for hierarchical knowledge integration in deep learning settings. For all applications, a faithful embedding of trees is needed, with combinatorial constructions emerging as the most effective direction. This paper identifies and solves two key limitations of existing works. First, the combinatorial construction hinges on finding highly separated points on a hypersphere, a notoriously difficult problem. Current approaches achieve poor separation, degrading the quality of the corresponding hyperbolic embedding. We propose highly separated Delaunay tree embeddings (HS-DTE), which integrates angular separation in a generalized formulation of Delaunay embeddings, leading to lower embedding distortion. Second, low-distortion requires additional precision. The current approach for increasing precision is to use multiple precision arithmetic, which renders the embeddings useless on GPUs in deep learning settings. We reformulate the combinatorial construction using floating point expansion arithmetic, leading to superior embedding quality while retaining utility on accelerated hardware.

Low-distortion and GPU-compatible Tree Embeddings in Hyperbolic Space

TL;DR

Addressing the need for faithful, GPU-friendly hyperbolic embeddings of tree-structured data, the paper proposes HS-DTE, a highly separated Delaunay-based constructive embedding that optimizes hyperspherical separation, and HypFPE, a floating point expansion framework for high-precision computations on GPUs. HS-DTE uses a minimal angle maximization objective to place children on hyperspheres with improved separation, enabling lower distortion across dimensions; HypFPE provides ulp-nonoverlapping expansions and new routines to compute hyperbolic distances via acosh and atanh with enhanced precision. Empirically, HS-DTE outperforms existing constructive and optimization-based methods on complete m-ary and phylogenetic trees, and HypFPE further boosts accuracy when combined with HS-DTE or other constructive methods, while preserving GPU compatibility. The work also delivers theoretical error guarantees for FPE-based distance computations and releases software libraries for arbitrary-dimensional hyperbolic embeddings and GPU-optimized precision arithmetic, broadening the practical impact for hyperbolic neural networks.

Abstract

Embedding tree-like data, from hierarchies to ontologies and taxonomies, forms a well-studied problem for representing knowledge across many domains. Hyperbolic geometry provides a natural solution for embedding trees, with vastly superior performance over Euclidean embeddings. Recent literature has shown that hyperbolic tree embeddings can even be placed on top of neural networks for hierarchical knowledge integration in deep learning settings. For all applications, a faithful embedding of trees is needed, with combinatorial constructions emerging as the most effective direction. This paper identifies and solves two key limitations of existing works. First, the combinatorial construction hinges on finding highly separated points on a hypersphere, a notoriously difficult problem. Current approaches achieve poor separation, degrading the quality of the corresponding hyperbolic embedding. We propose highly separated Delaunay tree embeddings (HS-DTE), which integrates angular separation in a generalized formulation of Delaunay embeddings, leading to lower embedding distortion. Second, low-distortion requires additional precision. The current approach for increasing precision is to use multiple precision arithmetic, which renders the embeddings useless on GPUs in deep learning settings. We reformulate the combinatorial construction using floating point expansion arithmetic, leading to superior embedding quality while retaining utility on accelerated hardware.

Paper Structure

This paper contains 37 sections, 9 theorems, 44 equations, 2 figures, 12 tables, 13 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and related work
  3. Hyperbolic geometry preliminaries
  4. Related work
  5. Hyperbolic tree embedding algorithms.
  6. Deep learning with hyperbolic tree embeddings.
  7. Floating point expansions.
  8. HS-DTE
  9. Setting and objective.
  10. Constructive solution for hyperbolic embeddings.
  11. The difficulty of distributing points on a hypersphere.
  12. Delaunay tree embeddings with separation.
  13. HypFPE: High-precision GPU-compatible hyperbolic embeddings
  14. Applying FPEs to the construction.
  15. The inverse hyperbolic cosine formulation
  16. ...and 22 more sections

Key Result

Theorem 3.1

The worst-case number of optimizations $p$ that has to be performed when embedding a tree with the combinatorial construction in Algorithm alg:sarkar with any objective using caching is

Figures (2)

  • Figure 1: Ablation studies on our construction and floating point expansion. (a) Minimal pairwise angle ($\uparrow$) of the hyperspherical points generated in step \ref{['ln:hyperspherical_gen']} of Algorithm \ref{['alg:sarkar']} using the various generation methods. The dimension of the space is set to 8, so the Hadamard method cannot generate more than 8 points. The MAM objective consistently leads to a higher separation angle. (b) The worst-case distortion ($\downarrow$, $D_{wc}$) of the constructed embedding of the phylogenetic tree with the maximal admissable $\tau$ given the number of bits. The vertical dashed line shows the limit with standard GPU floating point formats (float64). The horizontal dashed line is the best possible result $D_{wc} = 1$. FPE representations are required to get high quality embeddings without losing GPU-compatibility.
  • Figure 2: Pairwise relative distortions of h-MDS (top) and HS-DTE (bottom) applied to the 5-ary tree with a scaling factor $\tau = 5.0$. Axes are ordered using a breadth-first search of the tree.

Theorems & Definitions (20)

  • Theorem 3.1
  • proof
  • Definition 4.1
  • Theorem 4.2
  • proof
  • Proposition 4.3
  • proof
  • Theorem 4.4
  • proof
  • Proposition 4.5
  • ...and 10 more