Why are hyperbolic neural networks effective? A study on hierarchical representation capability

TL;DR

This work interrogates whether Hyperbolic Neural Networks truly realize the theoretical Hierarchical Representation Capability in hyperbolic space. It introduces the Hierarchical Representation Capability Benchmark ($ ext{HRCB}$) to quantify $HRC$ via metrics $M_r$, $M_o$, $M_p$, and $M_b$, and analyzes factors across manifold spaces and hierarchical structures using large-scale experiments and significance tests. The authors show that $HNN$s do not reach the hyperbolic upper limit; $HRC$ is shaped by optimization objectives and hierarchical structure, and they propose three pre-training strategies (EfD, ED, EfED) to enhance $HRC$ and downstream task performance within applicable scopes. The study provides a principled framework to evaluate $HRC$, reveals design guidelines for $HNN$s in hierarchical data, and demonstrates practical gains through targeted pre-training.

Abstract

Hyperbolic Neural Networks (HNNs), operating in hyperbolic space, have been widely applied in recent years, motivated by the existence of an optimal embedding in hyperbolic space that can preserve data hierarchical relationships (termed Hierarchical Representation Capability, HRC) more accurately than Euclidean space. However, there is no evidence to suggest that HNNs can achieve this theoretical optimal embedding, leading to much research being built on flawed motivations. In this paper, we propose a benchmark for evaluating HRC and conduct a comprehensive analysis of why HNNs are effective through large-scale experiments. Inspired by the analysis results, we propose several pre-training strategies to enhance HRC and improve the performance of downstream tasks, further validating the reliability of the analysis. Experiments show that HNNs cannot achieve the theoretical optimal embedding. The HRC is significantly affected by the optimization objectives and hierarchical structures, and enhancing HRC through pre-training strategies can significantly improve the performance of HNNs.

Figures (17)

• Figure 1: In theory, there exists an optimal embedding for hierarchical data in hyperbolic space, but HNNs can be affected by various factors and may not necessarily achieve the optimal embedding. Therefore, the effectiveness of HNNs cannot simply be attributed to the HRC of hyperbolic spaces.
• Figure 2: Four examples to help improve the four evaluation metrics ($\mathrm{M}_r,\mathrm{M}_o,\mathrm{M}_p,\mathrm{M}_b$). The range of the four evaluation metrics is between 0 and 1, the larger the better.
• Figure 3: Examples of horizontal hierarchical difference ($I_B$): it is clear that Figure (a) is more "unbalanced" than Figure (b), so the $I_B$ of Figure (a) is larger. Examples of vertical degree distribution ($I_D$): the closer to the root node in Figure (c), the greater the degree, so $I_D>0.5$, and vice versa, as in Figure (d), $I_D<0.5$.
• Figure 4: Three pre-training strategies for enhancing HRC. (1) EfD: Apply the HRC-enhanced encoder directly to downstream tasks while freezing its parameters. (2) ED: Apply the HRC-enhanced encoder directly to downstream tasks without freezing its parameters. (3) EfED: Place the HRC-enhanced encoder before the downstream task's encoder and freeze its parameters.
• Figure 5: Friedman test and Nemenyi post-hoc test for eleven methods ($\{\mathbb{R},\mathbb{D},\mathbb{H}\}\times\{\text{GAT,GCN,MLP}\}$+Comb+Comb(32) with 32-bit floating point precision). Comb has a floating point precision of 3000 bits, while other HNN methods have a floating point precision of 32 bits. Significance level $\alpha$ is 0.05. We also performed significance tests for all evaluation metrics ($\mathrm{M}_r,\mathrm{M}_o,\mathrm{M}_p,\mathrm{M}_b$) related to the HRC, so that each method contains 192 experimental results (eight dimensions $\times\{\text{GD,HR,FD}\}\times\{\text{Animal,Disease}\}\times\{\mathrm{M}_r,\mathrm{M}_o,\mathrm{M}_p,\mathrm{M}_b\}$). We removed the LR (Logistic Regression, NC tasks) unrelated to HRC, and the rationale behind this decision is discussed in Section \ref{['sec:factors_influence']}. Where $>$ denotes significantly better than, = denotes not significantly better than.