Free-Grained Hierarchical Recognition

TL;DR

Real-world hierarchical recognition often faces mixed-granularity supervision, where training labels vary in detail across a fixed taxonomy. The authors propose free-grain learning and introduce ImageNet-3L and ImageNet-F to benchmark full taxonomy prediction under heterogeneous supervision, using CLIP-based pruning to simulate annotation variability. They develop Text-Attr and Taxon-SSL as semantic and visual guidance methods (with a joint option) that significantly improve full-path accuracy over traditional baselines across ImageNet-F, iNat21-mini-F, and CUB-F. The work delivers both a scalable benchmark and practical baselines, advancing robust hierarchical classification under realistic supervision constraints and suggesting directions for imbalance-aware enhancements.

Abstract

Hierarchical image classification predicts labels across a semantic taxonomy, but existing methods typically assume complete, fine-grained annotations, an assumption rarely met in practice. Real-world supervision varies in granularity, influenced by image quality, annotator expertise, and task demands; a distant bird may be labeled Bird, while a close-up reveals Bald eagle. We introduce ImageNet-F, a large-scale benchmark curated from ImageNet and structured into cognitively inspired basic, subordinate, and fine-grained levels. Using CLIP as a proxy for semantic ambiguity, we simulate realistic, mixed-granularity labels reflecting human annotation behavior. We propose free-grain learning, with heterogeneous supervision across instances. We develop methods that enhance semantic guidance via pseudo-attributes from vision-language models and visual guidance via semi-supervised learning. These, along with strong baselines, substantially improve performance under mixed supervision. Together, our benchmark and methods advance hierarchical classification under real-world constraints.

Figures (11)

• Figure 1: Images vary in semantic detail: Some support only coarse labels, others reveal fine-grained categories. We propose free-grain learning: Training a hierarchical classifier with supervision free to vary in granularity across examples, reflecting semantic ambiguity in real-world images.
• Figure 2: Inconsistent and Noisy Hierarchy of ImageNet WordNet. (Left:) The histogram of hierarchy depths shows that ImageNet classes range from 5 to 19 levels, with many exceeding 10, which hinders consistent evaluation. (Right:) Sample hierarchies illustrate that classes can have multiple paths of different depths: Minivan appears in four paths at depths 12–15, while Teddy bear exists only at depth 7. This imbalance and inconsistency in the hierarchy make it unclear which path should be considered correct, underscoring the difficulty of using the original WordNet hierarchy for training and evaluation.
• Figure 3: We curate ImageNet-3L as a benchmark for hierarchical classification.(a) The original ImageNet taxonomy is noisy and inconsistent, with imbalances, overlaps, and multiple paths, making it unsuitable for reliable evaluation. (b) We construct a coherent 3-level taxonomy, inspired by cognitive psychology rosch1976basic: basic for general recognition, subordinate for contextual specificity, and fine-grained for specialized distinctions.
• Figure 4: Our ImageNet-F captures real-world challenges, where fine-grained labels follow a long-tailed distribution and granularity varies with visual clarity.Top: The graph shows the proportion of basic, subordinate, and fine-grained labels per class (sorted by ID). Fine-grained labels are scarce on the left but increase toward the right, forming a long tail. This imbalance often causes models to overfit to basic-level features and miss subtle distinctions, underscoring the need for robust multi-level learning. Bottom: Arrow-marked samples illustrate how label pruning reflects difficulty. (Last Column): a distant bird is labeled at the basic level (Bird); one with visible wings and talons at the subordinate level (Bird of prey); and a close-up at the fine-grained level (Bald eagle).
• Figure 5: Overview of the proposed methods. (a) Text-Attr enriches feature representations using semantic cues from images, compensating for missing labels and capturing shared attributes across levels. (b) Taxon-SSL handles missing-level labels by treating them as unlabeled and learns from visual consistency through augmented views.