In-Context Learning with Noisy Labels
Junyong Kang, Donghyun Son, Hwanjun Song, Buru Chang
TL;DR
This work identifies the practical problem that labels in in-context learning demonstrations can be noisy, which degrades both accuracy and stability. It formalizes in-context learning with noisy labels, introduces four baseline strategies adapted from learning with noisy labels, and presents a novel rectification method that fine-tunes a GPT-2 family model to correct demonstration labels using contextual information gathered across multiple examples. Empirical results across MRPC, SST-5, and Tweet datasets show the rectification approach robustly defends performance against label noise and outperforms the baselines across several LLMs and retrievers, with additional gains in stability and data efficiency. The study highlights an important and practical direction for maintaining reliable in-context learning in real-world settings where labeling noise is common and often unavoidable.
Abstract
In-context learning refers to the emerging ability of large language models (LLMs) to perform a target task without additional training, utilizing demonstrations of the task. Recent studies aim to enhance in-context learning performance by selecting more useful demonstrations. However, they overlook the presence of inevitable noisy labels in task demonstrations that arise during the labeling process in the real-world. In this paper, we propose a new task, in-context learning with noisy labels, which aims to solve real-world problems for in-context learning where labels in task demonstrations would be corrupted. Moreover, we propose a new method and baseline methods for the new task, inspired by studies in learning with noisy labels. Through experiments, we demonstrate that our proposed method can serve as a safeguard against performance degradation in in-context learning caused by noisy labels.