ACT: Agentic Classification Tree

TL;DR

ACT addresses the need for transparent AI in high-stakes settings by extending decision-tree reasoning to unstructured inputs through LLM-evaluated, language-based node prompts. It replaces rigid feature splits with a TextGrad-guided process that optimizes prompts to produce semantically meaningful partitions, balancing traditional impurity measures with semantic feedback. Empirical results on DIAGNO, SPAM, and JAILBREAK show ACT achieving competitive or superior accuracy to prompting baselines while delivering traceable, NL-based decision paths. This framework enables auditable, high-integrity decisions on text and other unstructured data, with potential for broader extensions to multi-class, regression, and multimodal contexts.

Abstract

When used in high-stakes settings, AI systems are expected to produce decisions that are transparent, interpretable, and auditable, a requirement increasingly expected by regulations. Decision trees such as CART provide clear and verifiable rules, but they are restricted to structured tabular data and cannot operate directly on unstructured inputs such as text. In practice, large language models (LLMs) are widely used for such data, yet prompting strategies such as chain-of-thought or prompt optimization still rely on free-form reasoning, limiting their ability to ensure trustworthy behaviors. We present the Agentic Classification Tree (ACT), which extends decision-tree methodology to unstructured inputs by formulating each split as a natural-language question, refined through impurity-based evaluation and LLM feedback via TextGrad. Experiments on text benchmarks show that ACT matches or surpasses prompting-based baselines while producing transparent and interpretable decision paths.

Figures (6)

• Figure 1: Example ACT decision tree for tuberculosis diagnosis using unstructured, free-text patient descriptions. A tree is automatically learned, with each node containing a binary natural language question, autonomously discovered via recursive prompt refinement to maximize label separation at each split. At inference, these questions are answered by a large language model (LLM) from the root node to the leaves of the tree. The final classification (TB or Not TB) corresponds to the majority label of training examples described by each leaf.
• Figure 2: Ablation study of ACT hyperparameters on the DIAGNO dataset using GPT-4.1 Nano. (a) Effect of depth parameter $d$ on model accuracy for different numbers of optimization steps per node during training $k$. (b) Effect of the number of optimization steps per node during training $k$ on model accuracy for different depth values $d$. Solid lines represent training accuracy; dashed lines represent test accuracy.
• Figure 3: Decision tree of depth 3 generated by the Gemma3-4B model for spam email classification. The tree distinguishes between spam and legitimate email (ham) through hierarchical semantic questions about promotional content, writing quality, and intent. Each internal node represents a binary question optimized through the ACT framework to maximize class separation, with leaf nodes indicating the final classification based on the majority class of training examples.
• Figure 4: Decision tree of depth 3 generated by the Qwen3-4b model for jailbreak prompt classification. The tree recursively partitions inputs through binary natural-language questions optimized to distinguish between jailbreak attempts (JB) and legitimate prompts (Not JB). Each internal node contains a semantically meaningful question discovered through iterative prompt refinement, with terminal nodes indicating the final classification based on the majority class of training examples reaching that leaf.
• Figure 5: Decision tree of depth 3 generated by training a CART model after performing a TF-IDF preprocessing step on the DIAGNO dataset.