HEAP: Unsupervised Object Discovery and Localization with Contrastive Grouping

TL;DR

HEAPAddress unsupervised object discovery and localization by building hierarchical relationships from patch to region to image level using a lightweight cross-attention head with learnable group tokens. It introduces three unsupervised losses—$L_{ ext{intra}}$ for intra-image grouping, $L_{ ext{neg}}$ for foreground-background disentanglement, and $L_{ ext{inter}}$ for inter-image clustering—to produce semantically coherent regions and discriminative foregrounds across images. The approach achieves state-of-the-art results on semantic segmentation retrieval, unsupervised object discovery, and saliency detection while remaining efficient, thanks to its encoder-free head and avoidance of extensive extra supervision. Overall, HEAP demonstrates that hierarchical contrastive grouping with cross-image context significantly improves object discovery across diverse datasets and tasks, with strong practical implications for weakly supervised learning and retrieval pipelines.

Abstract

Unsupervised object discovery and localization aims to detect or segment objects in an image without any supervision. Recent efforts have demonstrated a notable potential to identify salient foreground objects by utilizing self-supervised transformer features. However, their scopes only build upon patch-level features within an image, neglecting region/image-level and cross-image relationships at a broader scale. Moreover, these methods cannot differentiate various semantics from multiple instances. To address these problems, we introduce Hierarchical mErging framework via contrAstive grouPing (HEAP). Specifically, a novel lightweight head with cross-attention mechanism is designed to adaptively group intra-image patches into semantically coherent regions based on correlation among self-supervised features. Further, to ensure the distinguishability among various regions, we introduce a region-level contrastive clustering loss to pull closer similar regions across images. Also, an image-level contrastive loss is present to push foreground and background representations apart, with which foreground objects and background are accordingly discovered. HEAP facilitates efficient hierarchical image decomposition, which contributes to more accurate object discovery while also enabling differentiation among objects of various classes. Extensive experimental results on semantic segmentation retrieval, unsupervised object discovery, and saliency detection tasks demonstrate that HEAP achieves state-of-the-art performance.

Figures (3)

• Figure 1: (a)-(b): Existing methods, such as FOUND simeoni2023found and TokenCut wang2022self, consider patch-level similarities for each image separately. (a)-(c): HEAP jointly explores hierarchical supervisions (i.e., patch/region/image-level) across images, enabling more acurate object discovery and discrimination among objects of various classes. Note that colors only represent different objects, not specific classes.
• Figure 2: HEAP Overview. A pre-trained encoder (e.g., with DINO caron2021emerging) processes input images for patch embeddings. The cross-attention layers take in learnable group tokens and patch embeddings, and then adaptively aggregate image representations with each group token capturing distinct characteristics. Learned tokens cluster patches into regions based on embedding similarity. Patches belonging to the same regions are merged to form region-level embeddings, further aggregated for image-level foreground and background embeddings. HEAP is trained with three losses. Intra-image grouping: grouping patches based on similarities. Foreground-background contrasting: pushing foreground and background embeddings apart. Inter-image clustering: pulling similar regions closer, with the strength weighted by the similarity ranking. Note that $F^{f}_{i}$ represents the foreground embedding and $F^{b}_{i}$ represents the background embedding. $G_{i}$ can be $G^{f}_{i}$ or $G^{b}_{i}$, representing the region-level foreground or background embedding. We omit the process of obtaining $G^{f}_{i}$ and $G^{b}_{i}$ from region-level embeddings, which simplifies the illustration and does not affect understanding.
• Figure 3: Qualitative results of unsupervised object discovery and localization obtained by HEAP, FOUND simeoni2023found, and TokenCut wang2022self on VOC12. Our method could distinguish objects of multiple classes in a single image while recent works not. Note that colors only represent different objects in an image, not corresponding to specific classes.