Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-LLM Collaborative Feature Engineering for Tabular Data

Zhuoyan Li, Aditya Bansal, Jinzhao Li, Shishuang He, Zhuoran Lu, Mutian Zhang, Qin Liu, Yiwei Yang, Swati Jain, Ming Yin, Yunyao Li

TL;DR

A human-LLM collaborative feature engineering framework for tabular learning is proposed that improves feature engineering performance across a variety of tabular datasets and reduces users' cognitive load during the feature engineering process.

Abstract

Large language models (LLMs) are increasingly used to automate feature engineering in tabular learning. Given task-specific information, LLMs can propose diverse feature transformation operations to enhance downstream model performance. However, current approaches typically assign the LLM as a black-box optimizer, responsible for both proposing and selecting operations based solely on its internal heuristics, which often lack calibrated estimations of operation utility and consequently lead to repeated exploration of low-yield operations without a principled strategy for prioritizing promising directions. In this paper, we propose a human-LLM collaborative feature engineering framework for tabular learning. We begin by decoupling the transformation operation proposal and selection processes, where LLMs are used solely to generate operation candidates, while the selection is guided by explicitly modeling the utility and uncertainty of each proposed operation. Since accurate utility estimation can be difficult especially in the early rounds of feature engineering, we design a mechanism within the framework that selectively elicits and incorporates human expert preference feedback, comparing which operations are more promising, into the selection process to help identify more effective operations. Our evaluations on both the synthetic study and the real user study demonstrate that the proposed framework improves feature engineering performance across a variety of tabular datasets and reduces users' cognitive load during the feature engineering process.

Human-LLM Collaborative Feature Engineering for Tabular Data

TL;DR

A human-LLM collaborative feature engineering framework for tabular learning is proposed that improves feature engineering performance across a variety of tabular datasets and reduces users' cognitive load during the feature engineering process.

Abstract

Large language models (LLMs) are increasingly used to automate feature engineering in tabular learning. Given task-specific information, LLMs can propose diverse feature transformation operations to enhance downstream model performance. However, current approaches typically assign the LLM as a black-box optimizer, responsible for both proposing and selecting operations based solely on its internal heuristics, which often lack calibrated estimations of operation utility and consequently lead to repeated exploration of low-yield operations without a principled strategy for prioritizing promising directions. In this paper, we propose a human-LLM collaborative feature engineering framework for tabular learning. We begin by decoupling the transformation operation proposal and selection processes, where LLMs are used solely to generate operation candidates, while the selection is guided by explicitly modeling the utility and uncertainty of each proposed operation. Since accurate utility estimation can be difficult especially in the early rounds of feature engineering, we design a mechanism within the framework that selectively elicits and incorporates human expert preference feedback, comparing which operations are more promising, into the selection process to help identify more effective operations. Our evaluations on both the synthetic study and the real user study demonstrate that the proposed framework improves feature engineering performance across a variety of tabular datasets and reduces users' cognitive load during the feature engineering process.
Paper Structure (23 sections, 4 theorems, 38 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 23 sections, 4 theorems, 38 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Motivation and Problem Formulation
  5. Surrogate Model for Approximation of the Utility Function $g(\cdot)$
  6. Selection of LLM-Proposed Feature Operations When Human Expertise is Available
  7. Evaluations
  8. Experimental Setup
  9. Evaluation Results
  10. Analysis of Iterative Gains in Feature Engineering
  11. Analysis of Computational Scalability of the Proposed Method
  12. User Study: How Do Humans Perform and Perceive with Our Method?
  13. Conclusion
  14. Proof for Technical Lemmas in Section \ref{['sec:method']}
  15. Complete proof for Lemma \ref{['lem:confidence']}
  16. ...and 8 more sections

Key Result

Lemma 3.1

At round $t$, the LLM $M$ proposes a set of candidate operations $\mathcal{S}_t$. For any $\delta\in(0,1)$, with probability at least $1-\delta$, the deviation between the actual utility $g(e)$ and the predicted expected utility $\mu_t(e)$ is uniformly bounded for all $e\in \mathcal{S}_t$:

Figures (3)

  • Figure 1: Comparing the performance trajectories of the proposed method with two LLM-based baselines (CAAFE and OCTree) in the feature engineering process, using an iteration budget of 50 and MLP as the tabular learner across six tasks. Error shade indicates the standard error of the mean.
  • Figure 2: Comparing participants' average final feature engineering performance, completion time, and the user experience perceptions for the flight satisfaction prediction task in the Control, Sr, and our Alg treatment, respectively. Error bars represent the 95% confidence intervals of the mean values. $\textsuperscript{*}$, $\textsuperscript{**}$, and $\textsuperscript{***}$ denote statistical significance levels of $0.1$, $0.05$, and $0.01$ respectively.
  • Figure C.1: Comparing the performance trajectories of the proposed method with two LLM-based baselines (CAAFE and OCTree) in the feature engineering process, using an iteration budget of 50 and MLP as the tabular learner across seven tasks. Error shade indicates the standard error of the mean.

Theorems & Definitions (4)

  • Lemma 3.1
  • Lemma 3.2
  • Corollary 3.1
  • Lemma A.1