Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers

Gjergji Kasneci, Enkelejda Kasneci

TL;DR

Results indicate that integrating embeddings with traditional numerical and categorical features often enhances predictive performance, especially on datasets with class imbalance or limited features and samples, with improvements particularly notable in XGBoost and CatBoost classifiers.

Abstract

Feature engineering is crucial for optimizing machine learning model performance, particularly in tabular data classification tasks. Leveraging advancements in natural language processing, this study presents a systematic approach to enrich tabular datasets with features derived from large language model embeddings. Through a comprehensive ablation study on diverse datasets, we assess the impact of RoBERTa and GPT-2 embeddings on ensemble classifiers, including Random Forest, XGBoost, and CatBoost. Results indicate that integrating embeddings with traditional numerical and categorical features often enhances predictive performance, especially on datasets with class imbalance or limited features and samples, such as UCI Adult, Heart Disease, Titanic, and Pima Indian Diabetes, with improvements particularly notable in XGBoost and CatBoost classifiers. Additionally, feature importance analysis reveals that LLM-derived features frequently rank among the most impactful for the predictions. This study provides a structured approach to embedding-based feature enrichment and illustrates its benefits in ensemble learning for tabular data.

Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers

TL;DR

Results indicate that integrating embeddings with traditional numerical and categorical features often enhances predictive performance, especially on datasets with class imbalance or limited features and samples, with improvements particularly notable in XGBoost and CatBoost classifiers.

Abstract

Feature engineering is crucial for optimizing machine learning model performance, particularly in tabular data classification tasks. Leveraging advancements in natural language processing, this study presents a systematic approach to enrich tabular datasets with features derived from large language model embeddings. Through a comprehensive ablation study on diverse datasets, we assess the impact of RoBERTa and GPT-2 embeddings on ensemble classifiers, including Random Forest, XGBoost, and CatBoost. Results indicate that integrating embeddings with traditional numerical and categorical features often enhances predictive performance, especially on datasets with class imbalance or limited features and samples, such as UCI Adult, Heart Disease, Titanic, and Pima Indian Diabetes, with improvements particularly notable in XGBoost and CatBoost classifiers. Additionally, feature importance analysis reveals that LLM-derived features frequently rank among the most impactful for the predictions. This study provides a structured approach to embedding-based feature enrichment and illustrates its benefits in ensemble learning for tabular data.

Paper Structure

This paper contains 38 sections, 1 equation, 12 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Feature Enrichment Framework for LLM Embeddings
  4. General Framework and Definitions
  5. Feature Enrichment Process
  6. Methodology for Classifier Evaluation and Ablation Study
  7. Experimental Setup
  8. Datasets.
  9. Data Preprocessing.
  10. Feature Engineering and Enrichment
  11. Embedding Generation and Dimensionality Reduction.
  12. Feature Selection.
  13. Classifier Selection and Training
  14. Classifiers.
  15. Hyperparameter settings.
  16. ...and 23 more sections

Figures (12)

  • Figure 1: Overview of the experimental workflow stages.
  • Figure 2: Performance metrics (Weighted/Balanced Accuracy, F1 Score, and ROC-AUC) for different feature subsets on multiple datasets (Pima Indians Diabetes, UCI Adult, UCI Heart Disease, and UCI Wine Quality) using Random Forest, XGBoost, and CatBoost classifiers. Each plot shows how different feature subsets influence classifier performance, with results sorted by feature subset to facilitate comparisons.
  • Figure 3: Total wins for each feature subset per classifier.
  • Figure 4: Random Forest top-10 features for the three best performing feature subsets on the UCI Adult dataset.
  • Figure 5: XGBoost top-10 features for the three best-performing feature subsets on the UCI Pima Indian Diabetes dataset.
  • ...and 7 more figures