Grid Jigsaw Representation with CLIP: A New Perspective on Image Clustering

TL;DR

This work addresses unsupervised image clustering by identifying core limitations in capturing internal image structure and fine-grained semantics. It introduces pGJR, a framework that combines CLIP-derived priors with a sequential Grid Jigsaw Representation to learn more informative, higher-level features while accelerating convergence. The approach integrates CLIP as a frozen visual extractor and augments it with a GJR module to refine representations for clustering, using an objective that blends instance discrimination and feature decorrelation. Empirical results across six benchmarks show faster convergence and competitive or superior clustering performance, with insights into when to favor CLIP-based pGJR versus CNN-based GJR for different dataset characteristics.

Abstract

Unsupervised representation learning for image clustering is essential in computer vision. Although the advancement of visual models has improved image clustering with efficient visual representations, challenges still remain. Firstly, existing features often lack the ability to represent the internal structure of images, hindering the accurate clustering of visually similar images. Secondly, finer-grained semantic labels are often missing, limiting the ability to capture nuanced differences and similarities between images. In this paper, we propose a new perspective on image clustering, the pretrain-based Grid Jigsaw Representation (pGJR). Inspired by human jigsaw puzzle processing, we modify the traditional jigsaw learning to gain a more sequential and incremental understanding of image structure. We also leverage the pretrained CLIP to extract the prior features which can benefit from the enhanced cross-modal representation for richer and more nuanced semantic information and label level differentiation. Our experiments demonstrate that using the pretrained model as a feature extractor can accelerate the convergence of clustering. We append the GJR module to pGJR and observe significant improvements on common-use benchmark datasets. The experimental results highlight the effectiveness of our approach in the clustering task, as evidenced by improvements in the ACC, NMI, and ARI metrics, as well as the super-fast convergence speed.

Figures (7)

• Figure 1: The difference between existing jigsaw strategy and human jigsaw puzzles. Fig. \ref{['fig:1']} (a) presents the existing jigsaw methods which almost focus on learning permutation and sorting with pixels. Human jigsaw puzzles first use one piece as a benchmark to consider the parts around it which is learning splicing and linking by prior image semantics understanding as shown in Fig. \ref{['fig:1']} (b).
• Figure 2: The framework illustration of proposed GJR where Fig. \ref{['fig:2']} (a) is pixel-based version and Fig. \ref{['fig:2']} (b) is feature-based one. There are both five steps about CNN Extraction, Block Division, Grid Division, Linear Regression and Reconstruction, but in different orders.
• Figure 3: The framework illustration of proposed pGJR. pGJR is consisted of CLIP-based representation structure and jigsaw supplement for image clustering. The representation learning includes jigsaw feature network in forward prorogation and clustering constraint optimization in backward propagation.
• Figure 4: Convergence efficiency and performance compared with GJR and pGJR on CIFAR-10.
• Figure 5: Visualization of clustering feature representations distribution compared with GJR and pGJR.