Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

$\text{H}^2$em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learning

Lin Li, Jiahui Li, Jiaming Lei, Jun Xiao, Feifei Shao, Long Chen

TL;DR

CZSL requires recognizing unseen state–object pairs, a task hampered by Euclidean embeddings that poorly reflect large hierarchical taxonomies. H^2em introduces hierarchical hyperbolic embeddings in the Lorentz model, paired with a taxonomic entailment loss and a discriminative alignment loss, plus a Hyperbolic Cross-Modal Attention module to enable instance-aware fusion. The framework maps Euclidean CLIP features into hyperbolic space, enforcing semantic and conceptual hierarchies while distinguishing semantically similar compositions, and achieves state-of-the-art results in both closed-world and open-world CZSL across MIT-States, UT-Zappos, and CGQA. This work demonstrates the effectiveness of hyperbolic geometry for scalable, fine-grained compositional reasoning and offers a robust basis for extending hierarchical hyperbolic learning to other multimodal reasoning tasks.

Abstract

Compositional zero-shot learning (CZSL) aims to recognize unseen state-object compositions by generalizing from a training set of their primitives (state and object). Current methods often overlook the rich hierarchical structures, such as the semantic hierarchy of primitives (e.g., apple fruit) and the conceptual hierarchy between primitives and compositions (e.g, sliced apple apple). A few recent efforts have shown effectiveness in modeling these hierarchies through loss regularization within Euclidean space. In this paper, we argue that they fail to scale to the large-scale taxonomies required for real-world CZSL: the space's polynomial volume growth in flat geometry cannot match the exponential structure, impairing generalization capacity. To this end, we propose H2em, a new framework that learns Hierarchical Hyperbolic EMbeddings for CZSL. H2em leverages the unique properties of hyperbolic geometry, a space naturally suited for embedding tree-like structures with low distortion. However, a naive hyperbolic mapping may suffer from hierarchical collapse and poor fine-grained discrimination. We further design two learning objectives to structure this space: a Dual-Hierarchical Entailment Loss that uses hyperbolic entailment cones to enforce the predefined hierarchies, and a Discriminative Alignment Loss with hard negative mining to establish a large geodesic distance between semantically similar compositions. Furthermore, we devise Hyperbolic Cross-Modal Attention to realize instance-aware cross-modal infusion within hyperbolic geometry. Extensive ablations on three benchmarks demonstrate that H2em establishes a new state-of-the-art in both closed-world and open-world scenarios. Our codes will be released.

$\text{H}^2$em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learning

TL;DR

CZSL requires recognizing unseen state–object pairs, a task hampered by Euclidean embeddings that poorly reflect large hierarchical taxonomies. H^2em introduces hierarchical hyperbolic embeddings in the Lorentz model, paired with a taxonomic entailment loss and a discriminative alignment loss, plus a Hyperbolic Cross-Modal Attention module to enable instance-aware fusion. The framework maps Euclidean CLIP features into hyperbolic space, enforcing semantic and conceptual hierarchies while distinguishing semantically similar compositions, and achieves state-of-the-art results in both closed-world and open-world CZSL across MIT-States, UT-Zappos, and CGQA. This work demonstrates the effectiveness of hyperbolic geometry for scalable, fine-grained compositional reasoning and offers a robust basis for extending hierarchical hyperbolic learning to other multimodal reasoning tasks.

Abstract

Compositional zero-shot learning (CZSL) aims to recognize unseen state-object compositions by generalizing from a training set of their primitives (state and object). Current methods often overlook the rich hierarchical structures, such as the semantic hierarchy of primitives (e.g., apple fruit) and the conceptual hierarchy between primitives and compositions (e.g, sliced apple apple). A few recent efforts have shown effectiveness in modeling these hierarchies through loss regularization within Euclidean space. In this paper, we argue that they fail to scale to the large-scale taxonomies required for real-world CZSL: the space's polynomial volume growth in flat geometry cannot match the exponential structure, impairing generalization capacity. To this end, we propose H2em, a new framework that learns Hierarchical Hyperbolic EMbeddings for CZSL. H2em leverages the unique properties of hyperbolic geometry, a space naturally suited for embedding tree-like structures with low distortion. However, a naive hyperbolic mapping may suffer from hierarchical collapse and poor fine-grained discrimination. We further design two learning objectives to structure this space: a Dual-Hierarchical Entailment Loss that uses hyperbolic entailment cones to enforce the predefined hierarchies, and a Discriminative Alignment Loss with hard negative mining to establish a large geodesic distance between semantically similar compositions. Furthermore, we devise Hyperbolic Cross-Modal Attention to realize instance-aware cross-modal infusion within hyperbolic geometry. Extensive ablations on three benchmarks demonstrate that H2em establishes a new state-of-the-art in both closed-world and open-world scenarios. Our codes will be released.
Paper Structure (28 sections, 16 equations, 8 figures, 9 tables)

This paper contains 28 sections, 16 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. $\text{H}^2$em Framework
  6. Feature Extraction
  7. Euclidean-to-Hyperbolic Transfer.
  8. Training Objectives
  9. Experiments
  10. Experiment Settings
  11. Comparison with the State-of-the-Arts
  12. Comparison with Hyperbolic Methods
  13. Ablation Study
  14. Qualitative Analysis
  15. Conclusion
  16. ...and 13 more sections

Figures (8)

  • Figure 1: (a) Humans leverage dual hierarchical structures that exist among primitives (states and objects) and compositions for compositional recognition. (b) Models mimic hierarchical modeling in the flat geometry of Euclidean space distorts such tree-like structures, while the negative curvature of Hyperbolic space preserves the low-distortion embeddings.
  • Figure 2: Challenges of direct Euclidean-to-Hyperbolic transfer. (a) Specific concept and its general parent are in inverse hierarchy. (b) Semantically similar compositions are not sufficiently separated. (c) The desired hierarchical structure.
  • Figure 3: Overview of the $\text{H}^2$em framework. (a) Extracting image and text features in Euclidean space. (b) Projecting features into hyperbolic space to model their hierarchy. Hyperbolic Cross-Modal Attention (HCA) adaptively refines the text representations. (c) Jointly optimizing with a taxonomic entailment loss to enforce this hierarchy, and a discriminative alignment loss for cross-modal matching.
  • Figure 4: Ablation study (§\ref{['sec:ex_abla']}) on the coefficients $\beta_1$ and $\beta_2$ in training objective Eq. \ref{['eq:total']}.
  • Figure 5: Visualization results (§\ref{['supp:quali']}) of the visual embeddings in learned hyperbolic space leveraging randomly sampled samples from MIT-States isola2015discovering.
  • ...and 3 more figures