Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model Enhanced Machine Learning Estimators for Classification

Yuhang Wu, Yingfei Wang, Chu Wang, Zeyu Zheng

TL;DR

The paper tackles improving binary classification by integrating pre-trained large language models (LLMs) with classical ML estimators. It proposes three complementary strategies: (i) linear ensembles with fixed or adaptive weights to combine ML predictions and LLM signals, (ii) calibration and multi-accuracy approaches that incorporate LLM outputs to produce well-calibrated probabilities, and (iii) transfer-learning via LLM-driven data augmentation under covariate shift. Key contributions include the adaptive AdaLinear weighting, two calibration schemes (including a low-parameter, discretized approach), and a transfer-learning method that adds LLM-labeled target samples with a weakly supervised loss. Across four public NLP datasets (e.g., WANDS, Yelp, Emotion, Hate), the integrated methods consistently outperform either LLM or ML alone, demonstrating improved prediction accuracy and robustness to distribution changes, with practical implications for deploying LLM-enhanced classifiers in real-world tasks.

Abstract

Pre-trained large language models (LLM) have emerged as a powerful tool for simulating various scenarios and generating output given specific instructions and multimodal input. In this work, we analyze the specific use of LLM to enhance a classical supervised machine learning method for classification problems. We propose a few approaches to integrate LLM into a classical machine learning estimator to further enhance the prediction performance. We examine the performance of the proposed approaches through both standard supervised learning binary classification tasks, and a transfer learning task where the test data observe distribution changes compared to the training data. Numerical experiments using four publicly available datasets are conducted and suggest that using LLM to enhance classical machine learning estimators can provide significant improvement on prediction performance.

Large Language Model Enhanced Machine Learning Estimators for Classification

TL;DR

The paper tackles improving binary classification by integrating pre-trained large language models (LLMs) with classical ML estimators. It proposes three complementary strategies: (i) linear ensembles with fixed or adaptive weights to combine ML predictions and LLM signals, (ii) calibration and multi-accuracy approaches that incorporate LLM outputs to produce well-calibrated probabilities, and (iii) transfer-learning via LLM-driven data augmentation under covariate shift. Key contributions include the adaptive AdaLinear weighting, two calibration schemes (including a low-parameter, discretized approach), and a transfer-learning method that adds LLM-labeled target samples with a weakly supervised loss. Across four public NLP datasets (e.g., WANDS, Yelp, Emotion, Hate), the integrated methods consistently outperform either LLM or ML alone, demonstrating improved prediction accuracy and robustness to distribution changes, with practical implications for deploying LLM-enhanced classifiers in real-world tasks.

Abstract

Pre-trained large language models (LLM) have emerged as a powerful tool for simulating various scenarios and generating output given specific instructions and multimodal input. In this work, we analyze the specific use of LLM to enhance a classical supervised machine learning method for classification problems. We propose a few approaches to integrate LLM into a classical machine learning estimator to further enhance the prediction performance. We examine the performance of the proposed approaches through both standard supervised learning binary classification tasks, and a transfer learning task where the test data observe distribution changes compared to the training data. Numerical experiments using four publicly available datasets are conducted and suggest that using LLM to enhance classical machine learning estimators can provide significant improvement on prediction performance.
Paper Structure (14 sections, 23 equations, 1 figure, 2 tables, 3 algorithms)

This paper contains 14 sections, 23 equations, 1 figure, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem formulation and notation
  3. Integrating LLM into a Classical ML Estimator
  4. Linear combination of ML and LLM estimators
  5. Linear combination with adaptive weights
  6. Calibration with LLM
  7. Background on calibration
  8. Calibrate ML model with LLM
  9. Transfer learning with LLM
  10. Numerical experiments
  11. Datasets
  12. Combined estimators and calibration with LLM
  13. Transfer learning with LLM
  14. Conclusion

Figures (1)

  • Figure 1: t-SNE visualization of an illustration data set

Theorems & Definitions (2)

  • Definition 1
  • Definition 2