Hierarchical Semantic Alignment for Image Clustering

TL;DR

This work tackles image clustering by addressing noun-ambiguity in external semantic priors. It introduces CAE, a training-free method that constructs a semantic space from selected WordNet nouns and Flickr captions and aligns this space with image features via optimal transport, followed by adaptive fusion of image, noun, and caption modalities. The approach yields consistent improvements over prior training-free methods and demonstrates strong performance across multiple datasets, including ImageNet-1K, while also offering clear ablation insights and visualizations. Overall, CAE advances unsupervised clustering by leveraging complementary textual semantics through principled alignment and fusion, with potential for richer descriptions via future multimodal language models.

Abstract

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clustering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves clustering performance in a training-free manner. In our approach, we incorporate two complementary types of textual semantics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.

Figures (5)

• Figure 1: Observations of ambiguity in a single noun from the ImageNet ImageNet dataset. In (a), the words “crane” and "tank" can refer to entirely different objects, respectively. In (b), the semantic similar words "spaniel", "saluki", "truck", and "tractor" fail to distinguish the fine-grained classes.
• Figure 2: Comparison of embeddings similarity using nouns, captions, and our proposed method for two similar bird images ("Robin" and "Linnet"). Despite the high similarity in image embeddings (0.73), using only nouns or captions yields higher semantic similarity (0.64 and 0.56, respectively). By combining both nouns and captions, our method reduces the similarity score to 0.35, providing a more accurate distinction between the images.
• Figure 3: An overview of our method, which consists of two components, i.e., (a) Semantic Space Construction: Select the nouns and descriptions that include the same semantics with image embeddings. (b) Adaptive Semantics Fusion: The selected nouns and descriptions embeddings are leveraged to boost image semantics through adaptive fusion.
• Figure 4: Analysis of clustering performance by varying the number of image semantic centers on (a) UCF-101 and (b) ImageNet-Dogs datasets, respectively.
• Figure 5: Visualization of embeddings used for clustering on ImageNet-Dogs dataset. a) image embeddings from CLIP. b) noun embeddings by Eq. \ref{['eq:noun_counterpart']}. c) description embeddings by Eq. \ref{['eq:des_counterpart']}. d) embeddings by our method.

Theorems & Definitions (2)

• Theorem 1
• proof