LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers

TL;DR

This work addresses automating feature engineering for tabular data by introducing LLM-FE, a framework that uses LLMs as knowledge-guided evolutionary optimizers to generate feature transformation programs. It formulates feature generation as a bilevel optimization where $T = \pi_\theta(\mathcal{X}_{train})$ and $f^* = \arg\min_f \mathcal{L}_f(f(\mathcal{T}(\mathcal{X}_{tr})), \mathcal{Y}_{tr})$, aiming to maximize $\mathcal{E}$ on $\mathcal{T}(\mathcal{X}_{val})$. The method integrates structured prompts, memory-based in-context learning, data-driven evaluation, and a multi-population evolutionary strategy to explore the feature space efficiently. Empirical results across 19 classification and 10 regression datasets show that LLM-FE consistently improves predictive performance for models like XGBoost, MLP, and TabPFN, often with favorable computation. The findings underscore the value of combining domain knowledge, evolutionary refinement, and data-driven feedback to produce impactful, interpretable features for tabular learning tasks.

Abstract

Automated feature engineering plays a critical role in improving predictive model performance for tabular learning tasks. Traditional automated feature engineering methods are limited by their reliance on pre-defined transformations within fixed, manually designed search spaces, often neglecting domain knowledge. Recent advances using Large Language Models (LLMs) have enabled the integration of domain knowledge into the feature engineering process. However, existing LLM-based approaches use direct prompting or rely solely on validation scores for feature selection, failing to leverage insights from prior feature discovery experiments or establish meaningful reasoning between feature generation and data-driven performance. To address these challenges, we propose LLM-FE, a novel framework that combines evolutionary search with the domain knowledge and reasoning capabilities of LLMs to automatically discover effective features for tabular learning tasks. LLM-FE formulates feature engineering as a program search problem, where LLMs propose new feature transformation programs iteratively, and data-driven feedback guides the search process. Our results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines, significantly enhancing the performance of tabular prediction models across diverse classification and regression benchmarks.

Figures (13)

• Figure 1: Overview of the LLM-FE Framework. For a given dataset, LLM-FE follows these steps: (a) New Feature Generation, where an LLM generates feature transformation hypotheses as programs for a given tabular dataset; (b) Feature Engineering, where the feature transformation program is applied to the underlying dataset, resulting in a modified dataset; (c) Feature Evaluation, where the modified dataset with the new features is evaluated using a prediction model; (d) Experience Management, which maintains a buffer of high-scoring programs that act as in-context samples for LLM's iterative refinement prompt. The features generated by LLM-FE are interpretable, using LLM's knowledge of the problem's domain.
• Figure 2: Aggregated ablation study results across classification datasets, showcasing the impact of individual components on LLM-FE's performance: (a) Data Examples, (b) Domain Knowledge, and (c) Evolutionary Refinement. Values are normalized with respect to the base LLM-FE model to facilitate fair comparison across conditions.
• Figure 3: Qualitative Analysis on Impact of Domain Knowledge. illustrating how LLM-FE (b) utilizes domain knowledge to create meaningful features with descriptions, in contrast to feature engineering without domain insights (a) leading to uninterpretable outputs.
• Figure 4: Quantitative impact of domain knowledge on model accuracy. Using domain knowledge boosts performance compared to both the base model and LLM-FE without domain knowledge.
• Figure 5: Performance Trajectory Analysis. for LLM-FE w/o evolutionary refinement and LLM-FE. LLM-FE demonstrates a better trajectory, highlighting the advantage of evolutionary refinement.