Semantic Tree Inference on Text Corpa using a Nested Density Approach together with Large Language Model Embeddings

TL;DR

This work tackles the challenge of extracting hierarchical semantic structure from large text corpora represented by LLM embeddings. It introduces a nested density-based clustering framework, building multi-scale trees by progressively relaxing density thresholds through adaptive DBSCAN, and uses PCA-reduced $L_2$ distances rather than full-space cosine distance to improve structural fidelity. The pipeline includes automated node annotation with a large-context LLM and cross-domain evaluations on benchmarks (e.g., 20 Newsgroups, IMDB, AG News, Theses-fr-en) and institutional corpora (TU Wien, AUB), revealing coherent topic hierarchies and institution-specific landscapes. The approach enables data-driven, interpretable knowledge mapping and has potential applications in scientometrics and taxonomy refinement, though it faces challenges in labeling stability and dependence on embedding capacity.

Abstract

Semantic text classification has undergone significant advances in recent years due to the rise of large language models (LLMs) and their high dimensional embeddings. While LLM-embeddings are frequently used to store and retrieve text by semantic similarity in vector databases, the global structure semantic relationships in text corpora often remains opaque. Herein we propose a nested density clustering approach, to infer hierarchical trees of semantically related texts. The method starts by identifying texts of strong semantic similarity as it searches for dense clusters in LLM embedding space. As the density criterion is gradually relaxed, these dense clusters merge into more diffuse clusters, until the whole dataset is represented by a single cluster -- the root of the tree. By embedding dense clusters into increasingly diffuse ones, we construct a tree structure that captures hierarchical semantic relationships among texts. We outline how this approach can be used to classify textual data for abstracts of scientific abstracts as a case study. This enables the data-driven discovery research areas and their subfields without predefined categories. To evaluate the general applicability of the method, we further apply it to established benchmark datasets such as the 20 Newsgroups and IMDB 50k Movie Reviews, demonstrating its robustness across domains. Finally we discuss possible applications on scientometrics, topic evolution, highlighting how nested density trees can reveal semantic structure and evolution in textual datasets.

Figures (9)

• Figure 1: Tree building from a dataset represented by its embedding space. On the left, an embedding space with clusters of different densities is shown. One can see that the orange, and grape colored clusters embed into a larger blue ensemble. A conceptual tree built from such a two dimensional embedding space is shown on the right.
• Figure 2: Shematic represantation of the tree node / cluster annotation process
• Figure 3: A comparison of trees built from the 20 Newsgroups dataset under different distance measures
• Figure 4: Representations of the tree obtained from the 20 Newsgroups dataset. Trees were built from the first two principal components obtained from Qwen3-Embedding-8B qwen3embedding embeddings. Parameters used to build tree were: initial $\epsilon = 0.0039$, $\Delta\epsilon = 0.00001$, $\mathrm{minpts} = 5$.
• Figure 5: The IMDB 50K Reviews dataset colored by sentiments: The tree was built using the first two principal components derived from SFR-Embedding-Mistral SFRembedding embeddings. Parameters yield to the tree building algorithm were: initial $\epsilon = 0.5$, $\Delta\epsilon = 0.0001$, $\mathrm{minpts} = 5$.