Corrective In-Context Learning: Evaluating Self-Correction in Large Language Models

TL;DR

Corrective In-Context Learning (CICL) investigates self-correction in large language models by incorporating the model’s initial predictions and their ground-truth corrections into the in-context prompt for text classification. The approach defines a second round of ICL that uses feedback triplets to produce a corrected label $\tilde{y}$ for a given input, but across 17 datasets and four LLMs with $k=8$ and varying correction proportions, CICL consistently underperforms standard ICL, degrading as corrections increase. The results indicate that this in-context feedback introduces confusion and disrupts task understanding rather than refining predictions, challenging the viability of simple self-correction prompts in few-shot settings and suggesting the need for alternative strategies or prompt designs for robust in-context learning. The study also shows that exposing harder misclassified examples in standard ICL does not reliably improve performance, highlighting limitations of difficulty-based example selection for few-shot learning tasks.

Abstract

In-context learning (ICL) has transformed the use of large language models (LLMs) for NLP tasks, enabling few-shot learning by conditioning on labeled examples without finetuning. Despite its effectiveness, ICL is prone to errors, especially for challenging examples. With the goal of improving the performance of ICL, we propose corrective in-context learning (CICL), an approach that incorporates a model's incorrect predictions alongside ground truth corrections into the prompt, aiming to enhance classification accuracy through self-correction. However, contrary to our hypothesis, extensive experiments on text classification tasks demonstrate that CICL consistently underperforms standard ICL, with performance degrading as the proportion of corrections in the prompt increases. Our findings indicate that CICL introduces confusion by disrupting the model's task understanding, rather than refining its predictions. Additionally, we observe that presenting harder examples in standard ICL does not improve performance, suggesting that example difficulty alone may not be a reliable criterion for effective selection. By presenting these negative results, we provide important insights into the limitations of self-corrective mechanisms in LLMs and offer directions for future research.

Figures (5)

• Figure 1: CICL prompt example. The model is tasked with predicting the correct label based on its own prediction, using examples that include both the prediction and its correction.
• Figure 2: Diagram of the CICL methodology. For each test instance $x$, $k$ few-shot examples $(x_1, y_1), \dots, (x_k, y_k)$ are selected. Standard ICL generates predictions $\hat{y}_1, \dots, \hat{y}_k$ for these examples, which are used to build the CICL prompt. This, combined with the initial ICL prediction $\hat{y}$ for $x$, allows the model to predict the corrected label $\tilde{y}$.
• Figure 3: Mean Macro-F1 ($\pm$ Std Dev) across all datasets for each model, comparing standard ICL (blue) and CICL (orange). As the proportion of corrected examples increases ($x$-axis), the performance of CICL decreases ($y$-axis).
• Figure 4: Prompt format for standard ICL, showing ground truth labels for $k$ examples.
• Figure 5: Prompt format for CICL, showing predicted and ground truth ("correct") labels for $k$ examples.