Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery

Zhenqi He, Yuanpei Liu, Kai Han

TL;DR

SEAL tackles Generalized Category Discovery by leveraging naturally occurring semantic hierarchies to jointly learn coarse-to-fine categories. It introduces three components—semantic-aware multi-task learning, Cross-Granularity Consistency (CGC) self-distillation, and Hierarchical Semantic-guided Soft Contrastive Learning (HSCL)—grounded in information-theoretic motivation that hierarchical supervision tightens bounds on the mutual information between representations and labels. The approach achieves state-of-the-art results on fine-grained benchmarks such as the Semantic Shift Benchmark (SSB) and Herbarium19, while generalizing to coarse-grained datasets, and demonstrates efficiency by keeping training overhead low and inference focused on the target granularity. These contributions advance open-world category discovery by exploiting semantic taxonomies without manual hierarchical design, enabling scalable and robust discovery of both known and novel categories in unlabelled data.

Abstract

This paper investigates the problem of Generalized Category Discovery (GCD). Given a partially labelled dataset, GCD aims to categorize all unlabelled images, regardless of whether they belong to known or unknown classes. Existing approaches typically depend on either single-level semantics or manually designed abstract hierarchies, which limit their generalizability and scalability. To address these limitations, we introduce a SEmantic-aware hierArchical Learning framework (SEAL), guided by naturally occurring and easily accessible hierarchical structures. Within SEAL, we propose a Hierarchical Semantic-Guided Soft Contrastive Learning approach that exploits hierarchical similarity to generate informative soft negatives, addressing the limitations of conventional contrastive losses that treat all negatives equally. Furthermore, a Cross-Granularity Consistency (CGC) module is designed to align the predictions from different levels of granularity. SEAL consistently achieves state-of-the-art performance on fine-grained benchmarks, including the SSB benchmark, Oxford-Pet, and the Herbarium19 dataset, and further demonstrates generalization on coarse-grained datasets. Project page: https://visual-ai.github.io/seal/

SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery

TL;DR

SEAL tackles Generalized Category Discovery by leveraging naturally occurring semantic hierarchies to jointly learn coarse-to-fine categories. It introduces three components—semantic-aware multi-task learning, Cross-Granularity Consistency (CGC) self-distillation, and Hierarchical Semantic-guided Soft Contrastive Learning (HSCL)—grounded in information-theoretic motivation that hierarchical supervision tightens bounds on the mutual information between representations and labels. The approach achieves state-of-the-art results on fine-grained benchmarks such as the Semantic Shift Benchmark (SSB) and Herbarium19, while generalizing to coarse-grained datasets, and demonstrates efficiency by keeping training overhead low and inference focused on the target granularity. These contributions advance open-world category discovery by exploiting semantic taxonomies without manual hierarchical design, enabling scalable and robust discovery of both known and novel categories in unlabelled data.

Abstract

This paper investigates the problem of Generalized Category Discovery (GCD). Given a partially labelled dataset, GCD aims to categorize all unlabelled images, regardless of whether they belong to known or unknown classes. Existing approaches typically depend on either single-level semantics or manually designed abstract hierarchies, which limit their generalizability and scalability. To address these limitations, we introduce a SEmantic-aware hierArchical Learning framework (SEAL), guided by naturally occurring and easily accessible hierarchical structures. Within SEAL, we propose a Hierarchical Semantic-Guided Soft Contrastive Learning approach that exploits hierarchical similarity to generate informative soft negatives, addressing the limitations of conventional contrastive losses that treat all negatives equally. Furthermore, a Cross-Granularity Consistency (CGC) module is designed to align the predictions from different levels of granularity. SEAL consistently achieves state-of-the-art performance on fine-grained benchmarks, including the SSB benchmark, Oxford-Pet, and the Herbarium19 dataset, and further demonstrates generalization on coarse-grained datasets. Project page: https://visual-ai.github.io/seal/
Paper Structure (31 sections, 25 equations, 4 figures, 16 tables, 1 algorithm)

This paper contains 31 sections, 25 equations, 4 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Category Discovery.
  4. Hierarchical Learning.
  5. Preliminary
  6. Method
  7. Theoretical Motivation
  8. SEAL: Semantic-Aware Hierarchical Learning for GCD
  9. Semantic-aware Hierarchical Learning
  10. Cross-Granularity Consistency Self Distillation
  11. Hierarchical Semantic-guided Soft Contrastive Learning
  12. Overall Objective
  13. Experiment
  14. Experimental Setup
  15. Main Results
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Comparison of SEAL and previous methods rastegar2023learnwang2023discover using hierarchical learning. (a) In previous attempts, several upper and lower levels as well as abstract concepts are defined around the ground-truth level, which may cause errors in the hierarchical structure. (b) In our method, we propose to utilise the semantic information at different levels to enhance the GCD performance.
  • Figure 2: Overview of the proposed SEAL framework.
  • Figure 3: Overview of Hierarchical Semantic-guided Soft contrastive learning.
  • Figure 4: $t$-SNE visualization of 20 classes randomly sampled from the Stanford Cars krause20133d dataset.