ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation
Yiming Xu, Yuan Yuan, Vijay Viswanathan, Graham Neubig
TL;DR
ClusterFusion reframes text clustering by making an LLM the clustering core, guided by embedding-based subset partition to respect context windows. The approach enables direct incorporation of domain knowledge and user preferences through prompting, achieving state-of-the-art results on standard benchmarks and substantial gains on domain-specific datasets without fine-tuning. Key insights include the importance of exemplar ordering for summarization quality, and the complementary roles of domain knowledge and user preferences in shaping topic extraction and assignment. The work also provides a cost-conscious, training-free alternative with data and results released to support reproducibility.
Abstract
Text clustering is a fundamental task in natural language processing, yet traditional clustering algorithms with pre-trained embeddings often struggle in domain-specific contexts without costly fine-tuning. Large language models (LLMs) provide strong contextual reasoning, yet prior work mainly uses them as auxiliary modules to refine embeddings or adjust cluster boundaries. We propose ClusterFusion, a hybrid framework that instead treats the LLM as the clustering core, guided by lightweight embedding methods. The framework proceeds in three stages: embedding-guided subset partition, LLM-driven topic summarization, and LLM-based topic assignment. This design enables direct incorporation of domain knowledge and user preferences, fully leveraging the contextual adaptability of LLMs. Experiments on three public benchmarks and two new domain-specific datasets demonstrate that ClusterFusion not only achieves state-of-the-art performance on standard tasks but also delivers substantial gains in specialized domains. To support future work, we release our newly constructed dataset and results on all benchmarks.