Harnessing Large Language Models as Post-hoc Correctors

TL;DR

This work presents LlmCorr, a training-free framework that uses fixed LLMs to post-hoc correct predictions from arbitrary ML models. By constructing a contextual knowledge database from training and validation data, retrieving relevant context via embedding-based similarity, and prompting an LLM with carefully engineered templates (including a self-correction mechanism), LlmCorr improves predictive accuracy on molecule graph tasks and NLP benchmarks, achieving up to 39% RMSE improvement. The approach is model- and data-agnostic, offering a cost-effective way to boost performance without retraining, though results depend on LLM capabilities, prompt design, and token limits. The study also provides extensive ablations and discusses limitations, ethical considerations, and avenues for extending the framework to broader domains.

Abstract

As Machine Learning (ML) models grow in size and demand higher-quality training data, the expenses associated with re-training and fine-tuning these models are escalating rapidly. Inspired by recent impressive achievements of Large Language Models (LLMs) in different fields, this paper delves into the question: can LLMs efficiently improve an ML's performance at a minimal cost? We show that, through our proposed training-free framework LlmCorr, an LLM can work as a post-hoc corrector to propose corrections for the predictions of an arbitrary ML model. In particular, we form a contextual knowledge database by incorporating the dataset's label information and the ML model's predictions on the validation dataset. Leveraging the in-context learning capability of LLMs, we ask the LLM to summarise the instances in which the ML model makes mistakes and the correlation between primary predictions and true labels. Following this, the LLM can transfer its acquired knowledge to suggest corrections for the ML model's predictions. Our experimental results on text analysis and the challenging molecular predictions show that \model improves the performance of a number of models by up to 39%.

Figures (14)

• Figure 1: Harnessing LLMs as post-hoc correctors. A fixed LLM is leveraged to propose corrections to an arbitrary ML model's predictions without additional training or the need for additional datasets.
• Figure 2: A high-level overview of LlmCorr, harnessing Large Language Models (LLMs) as post-hoc correctors to refine predictions made by an arbitrary Machine Learning (ML) model.
• Figure 3: LlmCorr prompt template. Multiple contextual knowledge from training and validation datasets can be included by expanding the template.
• Figure 4: Self-correction prompt template.
• Figure 5: Ablation study of LlmCorr with a different number of contextual knowledge data ($k$) on ogbg-molbace and ogbg-molesol datasets.