Question-Driven Analysis and Synthesis: Building Interpretable Thematic Trees with LLMs for Text Clustering and Controllable Generation

TL;DR

This work addresses the interpretability gap in unsupervised text analysis by introducing Recursive Thematic Partitioning (RTP), an interactive framework that builds a binary tree of natural-language questions to partition a corpus. RTP produces interpretable, rule-based thematic hierarchies and demonstrates quantitative utility by using the trees as features for downstream classification, while also enabling Controllable Thematic Generation (CTG) through thematic-path prompts to guide text synthesis. The framework relies on a two-stage data reduction to manage context limits, recursive yes/no question generation, majority-vote partitioning, and stopping criteria that balance depth and leaf size. Empirical results on IMDB, Yelp, and AG News show RTP can outperform a strong keyword-based baseline in interpretability and, in some cases, in classification accuracy, with CTG providing principled control over generated content. Overall, RTP shifts text analysis from purely statistical discovery toward knowledge-driven, interactive modeling with practical implications for controlled generation and domain knowledge integration.

Abstract

Unsupervised analysis of text corpora is challenging, especially in data-scarce domains where traditional topic models struggle. While these models offer a solution, they typically describe clusters with lists of keywords that require significant manual effort to interpret and often lack semantic coherence. To address this critical interpretability gap, we introduce Recursive Thematic Partitioning (RTP), a novel framework that leverages Large Language Models (LLMs) to interactively build a binary tree. Each node in the tree is a natural language question that semantically partitions the data, resulting in a fully interpretable taxonomy where the logic of each cluster is explicit. Our experiments demonstrate that RTP's question-driven hierarchy is more interpretable than the keyword-based topics from a strong baseline like BERTopic. Furthermore, we establish the quantitative utility of these clusters by showing they serve as powerful features in downstream classification tasks, particularly when the data's underlying themes correlate with the task labels. RTP introduces a new paradigm for data exploration, shifting the focus from statistical pattern discovery to knowledge-driven thematic analysis. Furthermore, we demonstrate that the thematic paths from the RTP tree can serve as structured, controllable prompts for generative models. This transforms our analytical framework into a powerful tool for synthesis, enabling the consistent imitation of specific characteristics discovered in the source corpus.

Figures (5)

• Figure 1: Overview of tree generation for RTP. A random sample of the dataset is taken. Then, the sample is used within a prompt to generate a yes/no question capable of partitioning the sample. Using LLM calls, the question is answered for each item in the sample. The items are assigned to the "yes" leaf and the "no" leaf, and the question generation process is recursively performed in each leaf until stop conditions are met.
• Figure 2: Prompts used for summarization. The text data is provided to the LLM as a user prompt. By changing the max_words parameter, we can control the summary length. The JSON schema used for prompting helps parsing the answer.
• Figure 3: Prompts used for question generation. The text data is provided to the LLM as a user prompt containing a Python-style list of texts. The JSON schema used for prompting helps parsing the answer.
• Figure 4: Prompt used for data partitioning. The text data is provided to the LLM as a user prompt containing a Python-style list of texts. The JSON schema used for prompting helps parsing the answer.
• Figure 5: Prompt used for CTG. The JSON schema used for prompting helps parsing the answer.