Learning Structured Representations with Hyperbolic Embeddings

TL;DR

This work addresses the distortion that Euclidean spaces introduce when embedding hierarchical label structures into learned representations. It introduces HypStructure, a hyperbolic regularization framework comprising HypCPCC and HypCenter, which embeds the label hierarchy explicitly in hyperbolic space and is compatible with standard task losses. Empirical results across CIFAR10/100 and ImageNet100 show reduced hierarchy distortion, improved CPCC and classification performance, and enhanced OOD detection, with a formal eigenspectrum analysis linking hyperbolic geometry to these gains. The approach highlights the advantage of hyperbolic geometry for hierarchy-informed learning and offers practical implications for efficient, low-dimensional representations and reliable OOD detection.

Abstract

Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically. The code is available at \url{https://github.com/uiuctml/HypStructure}.

Figures (18)

• Figure 1: (left) An unweighted label tree with two coarse nodes: $F$, $G$. $F$ contains two fine classes $A,B$ and $G$ contains three fine classes $C,D,E$. We cannot embed this in $\ell_2$ exactly (right).
• Figure 1: Evaluation of hierarchical information distortion and classification accuracy using SupCon2020supcon as $\ell_\text{Flat}$. All metrics are reported as mean (standard deviation) over 3 seeds.
• Figure 2: Using $\ell_2$-CPCC for structured representation on CIFAR10. CIFAR10 hierarchy (left) has a three level structure with $13$ vertices. For a $512$-dimensional embedding, we apply $\ell_2$-CPCC either for the full tree (middle) or the leaf nodes only (right) and plot the ground truth tree metric against pairwise Euclidean centroid distances of the learnt representation. The optimal train CPCC is $1$.
• Figure 3: Lines on different models for $2$-dimensional hyperbolic space.
• Figure 4: Evaluation of distortion vs feature dimensions for HypStructure.

Theorems & Definitions (16)

• Definition 3.1: Manifold
• Definition 3.2: Poincaré Ball Model
• Definition 3.3: Klein Model
• Theorem 5.1: Eigenspectrum of Structured Representation with Balanced Label Tree
• Lemma A.1
• proof
• Corollary A.1
• proof
• Theorem A.1: Cadima_Calheiros_Preto_2010
• Lemma A.2