Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-Theoretic Generative Clustering of Documents

Xin Du, Kumiko Tanaka-Ishii

TL;DR

The paper introduces Generative Clustering (GC), a framework that clusters documents by transforming each document x into a distribution over texts Y generated by an LLM and measuring similarity via the KL divergence between p(Y|x) and cluster centroids p(Y|k). To handle the infinite Y, GC uses a regularized importance sampling (RIS) approach, with a second-moment-minimizing proposal φ and a two-step k-means-like iteration over sampled texts; the method converges to a local minimum of the distortion. Empirically, GC achieves state-of-the-art clustering performance across four datasets, demonstrates robustness to misspecified cluster counts, and improves generative retrieval when used to index documents hierarchically. The work highlights the value of leveraging rich, generative representations and information-theoretic objectives for unsupervised text clustering and retrieval applications.

Abstract

We present {\em generative clustering} (GC) for clustering a set of documents, $\mathrm{X}$, by using texts $\mathrm{Y}$ generated by large language models (LLMs) instead of by clustering the original documents $\mathrm{X}$. Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC achieves the state-of-the-art performance, outperforming any previous clustering method often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.

Information-Theoretic Generative Clustering of Documents

TL;DR

The paper introduces Generative Clustering (GC), a framework that clusters documents by transforming each document x into a distribution over texts Y generated by an LLM and measuring similarity via the KL divergence between p(Y|x) and cluster centroids p(Y|k). To handle the infinite Y, GC uses a regularized importance sampling (RIS) approach, with a second-moment-minimizing proposal φ and a two-step k-means-like iteration over sampled texts; the method converges to a local minimum of the distortion. Empirically, GC achieves state-of-the-art clustering performance across four datasets, demonstrates robustness to misspecified cluster counts, and improves generative retrieval when used to index documents hierarchically. The work highlights the value of leveraging rich, generative representations and information-theoretic objectives for unsupervised text clustering and retrieval applications.

Abstract

We present {\em generative clustering} (GC) for clustering a set of documents, , by using texts generated by large language models (LLMs) instead of by clustering the original documents . Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC achieves the state-of-the-art performance, outperforming any previous clustering method often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.

Paper Structure

This paper contains 36 sections, 3 theorems, 30 equations, 1 figure, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Document Clustering
  4. Generative Language Models
  5. Information-Theoretic Clustering
  6. Generative Clustering of Documents
  7. A Conventional Baseline Using BERT
  8. Generative Clustering Using Importance Sampling
  9. Two-Step Iteration Algorithm
  10. Distortion Function $d(x,k)$
  11. Proposal Distribution $\phi(Y)$
  12. Estimation of Cluster Centroid $p(Y|k)$.
  13. Probability Clipping
  14. Evaluation of Generative Clustering
  15. Data
  16. ...and 21 more sections

Key Result

Proposition 1

Algorithm alg:clustering converges to a local minimum of the total distortion: where $\hat{d}$ is defined in Eq. eq:ris.

Figures (1)

  • Figure 1: Performance of GC on the four datasets (columns) with varying (a-d) $\alpha$ values or (e-h) $J$ values. Different colors represent different clustering evaluation metrics, with error bars indicating the 95% confidence interval, based on 100 repeated experiments. Error bars for some $\alpha$ and $J$ values are too small to be visible.

Theorems & Definitions (6)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Proposition 3
  • proof