Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning

TL;DR

This work tackles the challenge of learning meaningful self-supervised graph representations in hyperbolic space, where hierarchical graphs are naturally modeled but conventional contrastive learning suffers Hyperbolic Dimensional Collapse ($HDC$). The authors propose HyperGCL, a framework that combines hyperbolic alignment with an outer-shell isotropy regularizer implemented via KL divergences on tangent-space distributions, ensuring level-wise uniformity along the outer shell of the Poincaré ball. Theoretical results show that radially symmetric tangent-space distributions map to isotropic ambient-space densities and that maximizing ERank aligns with the proposed regularizer, supporting stable, expressive representations. Empirically, HyperGCL achieves state-of-the-art performance on graph representation and collaborative-filtering benchmarks, with ablations confirming the necessity of the isotropy term and the influence of curvature and embedding dimension on hierarchical capacity and performance.

Abstract

Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity is measured by the cosine distance. However, many real-world graphs, especially of hierarchical nature, cannot be embedded well in the Euclidean space. Although the hyperbolic embedding is suitable for hierarchical representation learning, naively applying CL to the hyperbolic space may result in the so-called dimension collapse, i.e., features will concentrate mostly within few density regions, leading to poor utilization of the whole feature space. Thus, we propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information, as well as we propose a substitute of the uniformity metric to prevent the so-called dimensional collapse. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees. In the ambient space of the hyperbolic manifold these notions translate into imposing an isotropic ring density towards boundaries of Poincaré ball. Our experiments support the efficacy of our method.

Figures (9)

• Figure 1: Hypersphere vs. Hyperbola (viewed in the ambient space).
• Figure 3: Mapping isotropic Normal distribution from the tangent plane at $\mathbf{0}$ to the ambient space of the hyperbolic manifold results in the isotropic ring-shaped density, known as the Wrapped Normal distribution.
• Figure 4: An example of a low distortion embedding ( right) of a tree ( left) in the hyperbolic plane sarkar2011low where distances between direct neighbor nodes are preserved exactly, while non-neighbor distances enjoy at most $1\!+\!\varepsilon$ distortion factor.
• Figure 5: Several trees embedded in Poincaré disk. Fig. \ref{['fig:hyper_tree_1']}, \ref{['fig:hyper_tree_3']} & \ref{['fig:hyper_tree_4']} show tree embeddings under the dimensional collapse, whereas Fig. \ref{['fig:hyper_tree_2']} is a plausible "healthy" tree embedding (see text for details). Gray rings indicate the density of the underlying distribution.
• Figure 6: Our Hyperbolic Graph CL framework (HyperGCL).

Theorems & Definitions (11)

• Definition 2.1: Riemannian distance in $\mathbb{D}_c^d$
• Definition 2.2: Tangent Space
• Definition 2.3: Exponential/Logarithmic Map
• Theorem 4.1: Generalized Radial Distribution Mapping in Constant Curvature Spaces
• proof
• Theorem 4.2
• proof
• Definition 4.3: Effective Rank (ERank)
• Theorem 4.4: Convexity
• Theorem 4.5: Optima