Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies

TL;DR

This work tackles the difficulty of encoding data with latent hierarchies using discrete representations by shifting residual quantization from Euclidean to hyperbolic space. The authors introduce Hyperbolic Residual Quantization (HRQ) and HRQ-VAE, adapting embeddings, residual computation, and distance metrics to hyperbolic geometry to induce a hierarchy-friendly inductive bias. Empirical results on WordNet hypernym modeling and hierarchy discovery tasks show that HRQ-generated multitokens outperform Euclidean RQ by up to ~20% in hierarchy-related metrics and downstream tasks such as recommender systems. The findings suggest that integrating hyperbolic geometry into discrete hierarchical representations enhances the ability to capture latent hierarchical structure, with implications for knowledge graphs, taxonomy-driven NLP, and hierarchical recommendation.

Abstract

Hierarchical data arise in countless domains, from biological taxonomies and organizational charts to legal codes and knowledge graphs. Residual Quantization (RQ) is widely used to generate discrete, multitoken representations for such data by iteratively quantizing residuals in a multilevel codebook. However, its reliance on Euclidean geometry can introduce fundamental mismatches that hinder modeling of hierarchical branching, necessary for faithful representation of hierarchical data. In this work, we propose Hyperbolic Residual Quantization (HRQ), which embeds data natively in a hyperbolic manifold and performs residual quantization using hyperbolic operations and distance metrics. By adapting the embedding network, residual computation, and distance metric to hyperbolic geometry, HRQ imparts an inductive bias that aligns naturally with hierarchical branching. We claim that HRQ in comparison to RQ can generate more useful for downstream tasks discrete hierarchical representations for data with latent hierarchies. We evaluate HRQ on two tasks: supervised hierarchy modeling using WordNet hypernym trees, where the model is supervised to learn the latent hierarchy - and hierarchy discovery, where, while latent hierarchy exists in the data, the model is not directly trained or evaluated on a task related to the hierarchy. Across both scenarios, HRQ hierarchical tokens yield better performance on downstream tasks compared to Euclidean RQ with gains of up to $20\%$ for the hierarchy modeling task. Our results demonstrate that integrating hyperbolic geometry into discrete representation learning substantially enhances the ability to capture latent hierarchies.

Figures (4)

• Figure 1: Recall@10 of the hypernym generation based on tokens generated by HRQ vs tokens generated by RQ. HRQ consistently outperforms RQ. Furthermore, HRQ sustains consistent scores across different dimensionalities of the embedding.
• Figure 2: Visualization of the tangent space and related operations. Exponential map $exp_x^c$ maps from the tangent space attached at $x$ to the manifold and logarithmic map $log_x^c$ maps from the manifold to the tangent space attached at point $x$.
• Figure 3: HRQ-VAE visualized. In the image HRQ-VAE quantizes given vector $x$ into a multitoken $[t_0, t_1, t_2]$ and its corresponding embeddings $e_C^0, e_C^1, e_C^2$. Green blocks represent mapping to and from hyperbolic space. Yellow blocks represent hyperbolic autoencoder. The detailed part in the middle is responsible for hyperbolic residual quantization. The space expands exponentially the further we go away from the center. In fact, The circle's border is at infinite distance from point 0. As a consequence, most of the points must be distant from the center and only a small number of points can be at a privileged position close to the center. This leads to natural occurence of hierarchies. Light gray branches represent the possible HRQ-VAE and
• Figure 4: The embedding space structure induced by RQ-VAE and HRQ-VAE, respectively for a hierarchical tokens of length 2. The data is represented by coloured dots. Hue of the dot represents first hierarchical token. The shade represents second token. For the RQ-VAE the result is a typical effect of hierarchical clustering. HRQ-VAE due to exponential growth of the space has inductive bias to putting leaf nodes away on a similar distance away from the center.