Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models

Zihua Yang, Xin Liao, Yiqun Zhang, Yiu-ming Cheung

TL;DR

The paper tackles the semantic gap in categorical clustering caused by treating symbolic attribute values as equally distant and by weak co-occurrence signals in small datasets. It introduces ARISE, a framework that integrates external semantic knowledge from Large Language Models (LLMs) into the clustering pipeline via value-level queries, an attention-weighted encoding of LLM outputs, and an adaptive fusion with identity-based features to form discriminative representations for $k$-Means clustering. The method demonstrates consistent 19–27% gains over seven strong baselines across eight benchmark datasets using four LLM backends, while offline LLM description generation reduces online costs. The work highlights the practical potential of external semantic knowledge to enhance unsupervised categorical clustering and suggests avenues for extension to mixed-type data and domain-specific prompt tuning.

Abstract

Categorical data are prevalent in domains such as healthcare, marketing, and bioinformatics, where clustering serves as a fundamental tool for pattern discovery. A core challenge in categorical data clustering lies in measuring similarity among attribute values that lack inherent ordering or distance. Without appropriate similarity measures, values are often treated as equidistant, creating a semantic gap that obscures latent structures and degrades clustering quality. Although existing methods infer value relationships from within-dataset co-occurrence patterns, such inference becomes unreliable when samples are limited, leaving the semantic context of the data underexplored. To bridge this gap, we present ARISE (Attention-weighted Representation with Integrated Semantic Embeddings), which draws on external semantic knowledge from Large Language Models (LLMs) to construct semantic-aware representations that complement the metric space of categorical data for accurate clustering. That is, LLM is adopted to describe attribute values for representation enhancement, and the LLM-enhanced embeddings are combined with the original data to explore semantically prominent clusters. Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts, with gains of 19-27%. Code is available at https://github.com/develop-yang/ARISE

Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models

TL;DR

The paper tackles the semantic gap in categorical clustering caused by treating symbolic attribute values as equally distant and by weak co-occurrence signals in small datasets. It introduces ARISE, a framework that integrates external semantic knowledge from Large Language Models (LLMs) into the clustering pipeline via value-level queries, an attention-weighted encoding of LLM outputs, and an adaptive fusion with identity-based features to form discriminative representations for -Means clustering. The method demonstrates consistent 19–27% gains over seven strong baselines across eight benchmark datasets using four LLM backends, while offline LLM description generation reduces online costs. The work highlights the practical potential of external semantic knowledge to enhance unsupervised categorical clustering and suggests avenues for extension to mixed-type data and domain-specific prompt tuning.

Abstract

Categorical data are prevalent in domains such as healthcare, marketing, and bioinformatics, where clustering serves as a fundamental tool for pattern discovery. A core challenge in categorical data clustering lies in measuring similarity among attribute values that lack inherent ordering or distance. Without appropriate similarity measures, values are often treated as equidistant, creating a semantic gap that obscures latent structures and degrades clustering quality. Although existing methods infer value relationships from within-dataset co-occurrence patterns, such inference becomes unreliable when samples are limited, leaving the semantic context of the data underexplored. To bridge this gap, we present ARISE (Attention-weighted Representation with Integrated Semantic Embeddings), which draws on external semantic knowledge from Large Language Models (LLMs) to construct semantic-aware representations that complement the metric space of categorical data for accurate clustering. That is, LLM is adopted to describe attribute values for representation enhancement, and the LLM-enhanced embeddings are combined with the original data to explore semantically prominent clusters. Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts, with gains of 19-27%. Code is available at https://github.com/develop-yang/ARISE
Paper Structure (6 sections, 1 theorem, 1 equation, 2 figures, 1 table)

This paper contains 6 sections, 1 theorem, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Categorical Data Representation and Clustering
  4. Large Language Models for Clustering
  5. Proposed Method
  6. Semantic Representation Enrichment

Key Result

proposition thmcounterproposition

Computational Amortization. Let $\mathcal{C}_{query}$ be the unit cost of LLM querying. By operating on $\mathcal{V}$, the total extraction cost is $\mathcal{O}(|\mathcal{V}| \cdot \mathcal{C}_{query})$. Instance-level processing requires $N \times M$ queries (one per attribute-value pair), whereas

Figures (2)

  • Figure 1: The semantic gap in categorical clustering. (a) Non-semantic representation treats all values as equidistant ($d=1$), producing clusters with noticeable overlap. (b) Semantic-aware representation captures latent proximity: "oval" and "round" are highly similar ($d=0.2$), and "oval" is closer to "irregular" ($d=0.7$) than "round" is ($d=1$), yielding improved cluster separation.
  • Figure 2: Overview of ARISE. The framework integrates a semantic view (top) and an identity view (bottom). The semantic view enriches representations via structured prompting with an LLM followed by attention-weighted encoding. The identity view preserves categorical distinctions via identity encoding. Both views are fused through adaptive feature fusion, where the weight $\alpha^*$ is selected based on cluster quality, to produce discriminative representations for partitional clustering.

Theorems & Definitions (1)

  • proposition thmcounterproposition