Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Decision Tree Construction via Large Language Models

Lucas Carrasco, Felipe Urrutia, Andrés Abeliuk

TL;DR

The paper tackles learning from tabular data under data scarcity by constructing decision trees in a zero-shot fashion using large language models (LLMs). It frames tree construction as a sequence of LLM-driven steps: attribute splitting, probability estimation, and Gini impurity minimization, with harmonic-mean aggregation across branches and explicit prompts to avoid overconfidence. Experimental results show that zero-shot DTs built via LLMs can outperform a zero-shot TabLLM baseline and rival traditional trees in low-data settings, while maintaining interpretability. The work introduces a principled knowledge-driven baseline for zero-data tree construction, discusses biases and dependencies on pre-trained knowledge, and outlines practical avenues for refinement and fair, domain-specific deployment.

Abstract

This paper introduces a novel algorithm for constructing decision trees using large language models (LLMs) in a zero-shot manner based on Classification and Regression Trees (CART) principles. Traditional decision tree induction methods rely heavily on labeled data to recursively partition data using criteria such as information gain or the Gini index. In contrast, we propose a method that uses the pre-trained knowledge embedded in LLMs to build decision trees without requiring training data. Our approach leverages LLMs to perform operations essential for decision tree construction, including attribute discretization, probability calculation, and Gini index computation based on the probabilities. We show that these zero-shot decision trees can outperform baseline zero-shot methods and achieve competitive performance compared to supervised data-driven decision trees on tabular datasets. The decision trees constructed via this method provide transparent and interpretable models, addressing data scarcity while preserving interpretability. This work establishes a new baseline in low-data machine learning, offering a principled, knowledge-driven alternative to data-driven tree construction.

Zero-Shot Decision Tree Construction via Large Language Models

TL;DR

The paper tackles learning from tabular data under data scarcity by constructing decision trees in a zero-shot fashion using large language models (LLMs). It frames tree construction as a sequence of LLM-driven steps: attribute splitting, probability estimation, and Gini impurity minimization, with harmonic-mean aggregation across branches and explicit prompts to avoid overconfidence. Experimental results show that zero-shot DTs built via LLMs can outperform a zero-shot TabLLM baseline and rival traditional trees in low-data settings, while maintaining interpretability. The work introduces a principled knowledge-driven baseline for zero-data tree construction, discusses biases and dependencies on pre-trained knowledge, and outlines practical avenues for refinement and fair, domain-specific deployment.

Abstract

This paper introduces a novel algorithm for constructing decision trees using large language models (LLMs) in a zero-shot manner based on Classification and Regression Trees (CART) principles. Traditional decision tree induction methods rely heavily on labeled data to recursively partition data using criteria such as information gain or the Gini index. In contrast, we propose a method that uses the pre-trained knowledge embedded in LLMs to build decision trees without requiring training data. Our approach leverages LLMs to perform operations essential for decision tree construction, including attribute discretization, probability calculation, and Gini index computation based on the probabilities. We show that these zero-shot decision trees can outperform baseline zero-shot methods and achieve competitive performance compared to supervised data-driven decision trees on tabular datasets. The decision trees constructed via this method provide transparent and interpretable models, addressing data scarcity while preserving interpretability. This work establishes a new baseline in low-data machine learning, offering a principled, knowledge-driven alternative to data-driven tree construction.

Paper Structure

This paper contains 55 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Classification with LLMs
  4. Zero-Shot Classification
  5. Supervised Learning
  6. Fine-Tuning
  7. Interpretable Classification in Machine Learning
  8. Methodology
  9. Overview of the Process
  10. Iterative Tree Construction
  11. Attribute Splitting
  12. Numerical Attributes
  13. Categorical Attributes
  14. Probability Estimation
  15. Gini Impurity Minimization
  16. ...and 40 more sections

Figures (2)

  • Figure 1: Overview of Zero-Shot DT Construction via LLM: The input is information from the features (it is not necessary to have data for this, only knowledge of the variables to be used). The input is passed through the first 3 blocks that decide which feature will be used as a node to split the data into 2 branches. If the probabilities are greater than 0.9, or the depth is equal to or greater than the maximum depth, it is decided that the node is a leaf, and the label with the highest probability is given as a prediction; otherwise, the constraint imposed by the branch is added to a list of constraints, and iterates again, thus going down one level in the tree.
  • Figure 2: Comparison of decision trees: (Left) Zero-shot method with LLMs and (Right) CART algorithm for the Diabetes dataset. These trees are simplified versions where branches leading to unanimous outcomes (Yes or No) are condensed to single nodes.