ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering

TL;DR

ESMC tackles the need for user-driven, multi-criteria clustering by exploiting internal prompt-token embeddings from MLLMs. It identifies target embeddings via keyword-guided embedding localization and refines clustering with a lightweight two-layer MLP trained through pseudo-labels. Across seven diverse benchmarks, ESMC achieves competitive or superior clustering accuracy, with ablations confirming the value of the embedding-based targets and the clustering head, while maintaining efficiency and interpretability. The approach also highlights limitations tied to specific MLLM architectures and language priors, pointing to future work on broader applicability and stronger interpretability.

Abstract

Typical deep clustering methods, while achieving notable progress, can only provide one clustering result per dataset. This limitation arises from their assumption of a fixed underlying data distribution, which may fail to meet user needs and provide unsatisfactory clustering outcomes. Our work investigates how multi-modal large language models (MLLMs) can be leveraged to achieve user-driven clustering, emphasizing their adaptability to user-specified semantic requirements. However, directly using MLLM output for clustering has risks for producing unstructured and generic image descriptions instead of feature-specific and concrete ones. To address these issues, our method first discovers that MLLMs' hidden states of text tokens are strongly related to the corresponding features, and leverages these embeddings to perform clusterings from any user-defined criteria. We also employ a lightweight clustering head augmented with pseudo-label learning, significantly enhancing clustering accuracy. Extensive experiments demonstrate its competitive performance on diverse datasets and metrics.

Figures (10)

• Figure 1: Multiple clustering of sample images on the Stanford_cars dataset with cluster criterion of color, manufacturer, and body style.
• Figure 2: The ESMC framework consists of three key components: (i) the MLLM processes multimodal inputs (images and text prompts); (ii) target embeddings are selected from the language model's hidden states corresponding to text tokens; and (iii) a lightweight clustering head is trained to enhance cluster accuracy.
• Figure 3: The relationship between vocabulary logits and text prompt tokens, note that the color bar varies across the figures. The image is from the Stanford_cars dataset.
• Figure 4: Left: Two MLLM outputs with shared semantic meaning but distinct formatting. Right: The target embedding we choose $E_{25}(267)$, shows higher consistency in "sky" logit, as the clustering label in the environment criterion.
• Figure 5: Left: Conversation reveals that the MLLM response can be too general. It responds as "pink rose" instead of "rose" directly, while we only care about the species feature instead of the color feature. Right: Our target embedding $E_{25}(263)$ shows high logit in "rose" logit and low in "pink" logit, which could indicate the species feature.