Reliable Pseudo-labeling via Optimal Transport with Attention for Short Text Clustering

TL;DR

Short-text clustering is hindered by sparse semantic content, leading to weak representations. The paper introduces POTA, a three-component framework that fuses instance-level attention with a consistency-aware adaptive optimal transport (CAOT) to generate reliable pseudo-labels and drive cluster- and instance-level contrastive learning. CAOT jointly enforces sample-to-cluster global structure and sample-to-sample semantic consistency through entropy and semantic regularization, solved efficiently by a generalized conditional gradient method with Lagrange multipliers. Empirical results on eight benchmark datasets show POTA achieving state-of-the-art clustering performance and producing more discriminative representations, with robustness to data imbalance and improved pseudo-label reliability.

Abstract

Short text clustering has gained significant attention in the data mining community. However, the limited valuable information contained in short texts often leads to low-discriminative representations, increasing the difficulty of clustering. This paper proposes a novel short text clustering framework, called Reliable \textbf{P}seudo-labeling via \textbf{O}ptimal \textbf{T}ransport with \textbf{A}ttention for Short Text Clustering (\textbf{POTA}), that generate reliable pseudo-labels to aid discriminative representation learning for clustering. Specially, \textbf{POTA} first implements an instance-level attention mechanism to capture the semantic relationships among samples, which are then incorporated as a semantic consistency regularization term into an optimal transport problem. By solving this OT problem, we can yield reliable pseudo-labels that simultaneously account for sample-to-sample semantic consistency and sample-to-cluster global structure information. Additionally, the proposed OT can adaptively estimate cluster distributions, making \textbf{POTA} well-suited for varying degrees of imbalanced datasets. Then, we utilize the pseudo-labels to guide contrastive learning to generate discriminative representations and achieve efficient clustering. Extensive experiments demonstrate \textbf{POTA} outperforms state-of-the-art methods. The code is available at: \href{https://github.com/YZH0905/POTA-STC/tree/main}{https://github.com/YZH0905/POTA-STC/tree/main}.

Figures (7)

• Figure 1: Schematic illustration of the motivation. Conventional OT only considers the sample-to-cluster relationship, causing similar and adjacent samples to be assigned different pseudo-labels (red boxes). Our proposed CAOT addresses this issue by incorporating sample-to-sample semantic consistency, which will producing more accurate pseudo-labels
• Figure 2: Overall structure of POTA. Our model contains three components: (a) Pseudo-label Generation Module, (b) Semantic Similarity Construction Module and (c) Contrastive Learning Module.
• Figure 3: The structure of Attention Network. $\boldsymbol{S}^{(1)}$ denotes the similarity matrix among samples.
• Figure 4: T-SNE visualization of the representations on Stackoverflow, each color indicates ground truth category.
• Figure 5: The comparison of representation quality. The shaded regions represent the variance derived from 50 runs with different random seeds.