Text Clustering with Large Language Model Embeddings

TL;DR

The paper addresses how textual embeddings, particularly from large language models, and clustering algorithms influence text clustering performance across diverse datasets. It systematically compares TF‑IDF, BERT, and multiple OpenAI, Falcon, and LLaMA embeddings using a suite of clustering methods (k‑means, k‑means++, AHC, Spectral, Fuzzy C‑means) and evaluates with external and internal metrics, while also exploring dimensionality reduction via summarisation and the impact of larger model sizes. Key findings show that OpenAI embeddings generally yield superior clustering on structured text, with BERT leading among open models; summarisation does not consistently improve results, and larger models do not guarantee better performance, highlighting a trade-off between representation quality and computational cost. The study extends traditional clustering frameworks by integrating LLM embeddings, offering practical guidance for building scalable, effective text clustering pipelines and pointing to future work on diverse data types and model regimes.

Abstract

Text clustering is an important method for organising the increasing volume of digital content, aiding in the structuring and discovery of hidden patterns in uncategorised data. The effectiveness of text clustering largely depends on the selection of textual embeddings and clustering algorithms. This study argues that recent advancements in large language models (LLMs) have the potential to enhance this task. The research investigates how different textual embeddings, particularly those utilised in LLMs, and various clustering algorithms influence the clustering of text datasets. A series of experiments were conducted to evaluate the impact of embeddings on clustering results, the role of dimensionality reduction through summarisation, and the adjustment of model size. The findings indicate that LLM embeddings are superior at capturing subtleties in structured language. OpenAI's GPT-3.5 Turbo model yields better results in three out of five clustering metrics across most tested datasets. Most LLM embeddings show improvements in cluster purity and provide a more informative silhouette score, reflecting a refined structural understanding of text data compared to traditional methods. Among the more lightweight models, BERT demonstrates leading performance. Additionally, it was observed that increasing model dimensionality and employing summarisation techniques do not consistently enhance clustering efficiency, suggesting that these strategies require careful consideration for practical application. These results highlight a complex balance between the need for refined text representation and computational feasibility in text clustering applications. This study extends traditional text clustering frameworks by integrating embeddings from LLMs, offering improved methodologies and suggesting new avenues for future research in various types of textual analysis.

Figures (1)

• Figure 1: Representation of different embeddings for the CSTR dataset, where PCA was used as a preliminary dimensionality reduction algorithm and t-SNE for data projection into a lower-dimensional space.