Unsupervised Text Segmentation via Kernel Change-Point Detection on Sentence Embeddings

TL;DR

This work tackles unsupervised text segmentation by marrying pretrained sentence embeddings with kernel change-point detection in a training-free pipeline called Embed-KCPD. It advances theory by deriving dependence-aware guarantees for penalized KCPD under $m$-dependence, including an oracle inequality and a localization rate, and couples this with a practical algorithm solved efficiently via PELT. A key contribution is the empirical validation through both standard benchmarks and an LLM-based simulation framework that enforces controlled short-range dependence, demonstrating robust performance across embeddings and kernels. The combination of principled guarantees, simulation-based validation, and strong empirical results across diverse datasets positions Embed-KCPD as a competitive unsupervised baseline for text segmentation with practical applicability to real-world text streams.

Abstract

Unsupervised text segmentation is crucial because boundary labels are expensive, subjective, and often fail to transfer across domains and granularity choices. We propose Embed-KCPD, a training-free method that represents sentences as embedding vectors and estimates boundaries by minimizing a penalized KCPD objective. Beyond the algorithmic instantiation, we develop, to our knowledge, the first dependence-aware theory for KCPD under $m$-dependent sequences, a finite-memory abstraction of short-range dependence common in language. We prove an oracle inequality for the population penalized risk and a localization guarantee showing that each true change point is recovered within a window that is small relative to segment length. To connect theory to practice, we introduce an LLM-based simulation framework that generates synthetic documents with controlled finite-memory dependence and known boundaries, validating the predicted scaling behavior. Across standard segmentation benchmarks, Embed-KCPD often outperforms strong unsupervised baselines. A case study on Taylor Swift's tweets illustrates that Embed-KCPD combines strong theoretical guarantees, simulated reliability, and practical effectiveness for text segmentation.

Figures (11)

• Figure 1: Segmentation accuracies versus sequence length $T$ for Embed-KCPD applied to synthetically generated short-range dependent text data with GPT-4.1 and $m=20$. Curves compare three embedding methods (sBERT, MPNet, text-embedding-3-small, RoBERTa). Dashed red line shows the growth of the number of change points $K \approx 2\log T$.
• Figure 2: Timeline of Taylor Swift’s tweet stream segmented by Embed-KCPD using RBF and cosine kernels. Each segment is annotated with its tweet count (blue boxes) and an interpretation of its content (pink boxes).
• Figure 3: $P_k$ error (%) versus sequence length $T$ for Embed-KCPD applied to synthetically generated short-range dependent text data with GPT-4.1, $m=20$, for multiple values of $C$ and sBERT embeddings.
• Figure 4: $P_k$ error (%) versus sequence length $T$ for Embed-KCPD applied to synthetically generated short-range dependent text data with GPT-4.1, $C=0.1$, for multiple values of $m$ (number of sentences in LLM generation) and sBERT embeddings.
• Figure 5: Sensitivity of the number of detected segments to the hyperparameter $C$ on Wiki-300.

Theorems & Definitions (16)

• Lemma 4.9: uniform deviation over all segments
• Proposition 4.10: stability on homogeneous segments
• Theorem 4.11: oracle inequality
• Theorem 4.12: localization rate
• Proposition 1.1: m-dependent concentration for segment cost
• proof
• Lemma 1.2: Signal strength on a mixed segment
• proof
• Lemma 1.3: Detectability
• proof