Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, Jinwoo Shin

TL;DR

This work proposes a novel tabular learning framework that utilizes large language models (LLMs) to identify effective feature generation rules without manually specifying the search space and uses decision trees to convey this reasoning information, effectively providing knowledge from prior experiments to the LLMs.

Abstract

In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.

Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

TL;DR

This work proposes a novel tabular learning framework that utilizes large language models (LLMs) to identify effective feature generation rules without manually specifying the search space and uses decision trees to convey this reasoning information, effectively providing knowledge from prior experiments to the LLMs.

Abstract

In tabular prediction tasks, tree-based models combined with automated feature engineering methods often outperform deep learning approaches that rely on learned representations. While these feature engineering techniques are effective, they typically depend on a pre-defined search space and primarily use validation scores for feature selection, thereby missing valuable insights from previous experiments. To address these limitations, we propose a novel tabular learning framework that utilizes large language models (LLMs), termed Optimizing Column feature generator with decision Tree reasoning (OCTree). Our key idea is to leverage the reasoning capabilities of LLMs to identify effective feature generation rules without manually specifying the search space and provide language-based reasoning information highlighting past experiments as feedback for iterative rule improvements. We use decision trees to convey this reasoning information, as they can be easily represented in natural language, effectively providing knowledge from prior experiments (i.e., the impact of the generated features on performance) to the LLMs. Our empirical results demonstrate that OCTree consistently enhances the performance of various prediction models across diverse benchmarks, outperforming competing automated feature engineering methods. Code is available at https://github.com/jaehyun513/OCTree.
Paper Structure (30 sections, 2 equations, 4 figures, 20 tables)

This paper contains 30 sections, 2 equations, 4 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Optimizing feature generator with decision tree reasoning
  4. Tabular feature generation as rule generator optimization
  5. Generating column features with OCTree
  6. Experiments
  7. Main results: Context-aware feature engineering
  8. Main result: Context-agnostic feature engineering
  9. Ablations and analysis
  10. Conclusion
  11. Prompt examples
  12. Generation of a new column
  13. Rule initialization
  14. Generation of a rule
  15. Translation of a rule into Python code
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Overview of OCTree. (Step 0) Prompt the LLM to propose a name for the new column. (Step 1) Generate a rule by prompting the LLM with feedback on previously generated rules and relevant information for reasoning about the data. (Step 2) Generate a new column feature based on the proposed rule. (Step 3) Train a prediction model on the new data and compute the validation score and tree-based reasoning, provided as feedback for iterative improvements. (Step 4) Repeat steps 1-3 a fixed number of times and select the rule with the best validation score.
  • Figure 2: Generation of multiple features. The optimization process is repeated to generate multiple column features in sequence.
  • Figure 3: Imputing features with real data, i.e., Age. We report the mean accuracy (%) across three random splits on the Clinical dataset using XGBoost.
  • Figure 8: LLM identifies important features. We report the mean error (%) and standard deviation across three random splits on the Disease dataset. Both GPT-4o and Llama 2 identify the cough feature as more important, consistent with the accuracy seen in XGBoost models trained with and without these features.