Embedding Domain-Specific Knowledge from LLMs into the Feature Engineering Pipeline

TL;DR

This work tackles the high computational cost of evolutionary feature engineering by introducing a two-step pipeline that first leverages an LLM to suggest feature combinations using only feature names and the target objective, preserving data privacy, and then applies GP-based feature engineering (M3GP/M6GP) wrapped around Ridge/ RF models. The approach is evaluated on six regression and five classification datasets, showing GPT-based feature construction yields consistent gains on several well-studied problems, while GP-based wrappers can surpass strong baselines and produce compact models. The results demonstrate that integrating domain knowledge via LLMs with GP-driven feature construction can reduce computation and improve robustness, particularly for well-understood tasks, with M3GP and M6GP extending applicability to symbolic regression. Overall, the proposed pipeline offers a practical, privacy-preserving route to accelerate learning and achieve competitive performance with substantially smaller models than standard RF baselines.

Abstract

Feature engineering is mandatory in the machine learning pipeline to obtain robust models. While evolutionary computation is well-known for its great results both in feature selection and feature construction, its methods are computationally expensive due to the large number of evaluations required to induce the final model. Part of the reason why these algorithms require a large number of evaluations is their lack of domain-specific knowledge, resulting in a lot of random guessing during evolution. In this work, we propose using Large Language Models (LLMs) as an initial feature construction step to add knowledge to the dataset. By doing so, our results show that the evolution can converge faster, saving us computational resources. The proposed approach only provides the names of the features in the dataset and the target objective to the LLM, making it usable even when working with datasets containing private data. While consistent improvements to test performance were only observed for one-third of the datasets (CSS, PM, and IM10), possibly due to problems being easily explored by LLMs, this approach only decreased the model performance in 1/77 test cases. Additionally, this work introduces the M6GP feature engineering algorithm to symbolic regression, showing it can improve the results of the random forest regressor and produce competitive results with its predecessor, M3GP.

Figures (4)

• Figure 1: Proposed pipeline with two feature engineering steps. An LLM recommends feature combinations based on the available features and objectives, without accessing the dataset. These combinations are then added to the dataset, and M3GP or M6GP is used as a second feature engineering step.
• Figure 2: Median test RMSE over 100 generations when using M3GP-Ridge and M6GP-Ridge in the CSS and IS datasets. The plots highlight the space between quartiles 1 and 3, showing a large dispersion of values when inducing models using IS.
• Figure 3: Median test RMSE over 30 generations when using M6GP-RF and M6GP-Ridge in the HL and PT datasets, respectively. The plots highlight the space between quartiles 1 and 3. These plots are representative of the other datasets.
• Figure 4: Median test WAF over 30 generations when using M6GP-Ridge and M6GP-RF in the SAS dataset. The plots highlight the space between quartiles 1 and 3.