Memorization in In-Context Learning

TL;DR

The paper tackles the problem of understanding why in-context learning (ICL) improves performance by examining memorization of training data and its relationship to downstream results. It adapts a data-contamination memorization detection method to k-shot ICL prompts, using three prompt settings and a replication-based evaluation with exact and near-exact matches to surface memorized instances. The authors demonstrate that memorization surfaces strongly in ICL, especially with demonstrations lacking labels, and that performance gains correlate with surfaced memorization when ICL outperforms zero-shot, with thresholds around $40\%$ memorization linked to gains across multiple datasets. The work highlights practical implications for generalization, privacy, and safety in LLM prompting and motivates further study on distinguishing memorization from genuine generalization in demonstrations.

Abstract

In-context learning (ICL) has proven to be an effective strategy for improving the performance of large language models (LLMs) with no additional training. However, the exact mechanism behind this performance improvement remains unclear. This study is the first to show how ICL surfaces memorized training data and to explore the correlation between this memorization and performance on downstream tasks across various ICL regimes: zero-shot, few-shot, and many-shot. Our most notable findings include: (1) ICL significantly surfaces memorization compared to zero-shot learning in most cases; (2) demonstrations, without their labels, are the most effective element in surfacing memorization; (3) ICL improves performance when the surfaced memorization in few-shot regimes reaches a high level (about 40%); and (4) there is a very strong correlation between performance and memorization in ICL when it outperforms zero-shot learning. Overall, our study uncovers memorization as a new factor impacting ICL, raising an important question: to what extent do LLMs truly generalize from demonstrations in ICL, and how much of their success is due to memorization?

Figures (5)

• Figure 1: Illustrative examples of a two-shot ICL prompt for replicating instances from NLI (left) and classification (right) tasks. Note that, in our actual experiments, we use $k$-shot ICL, where $k = \{0, 25, 50, 100, 200\}$. All colored segments, except the green one, form the input prompt. Specifically, the gray segments indicate the instruction, the red segments display the two demonstrations, the blue segments correspond to the dataset instance being replicated, and the green segment exhibits the generated completion by the underlying LLM (GPT-4) for the subsequent segment of the dataset instance being replicated. For both examples, the generated completions are exact matches.
• Figure 2: Results on quantifying memorization in different ICL regimes for all settings. Plots on the left display memorization using exact and near‐exact matches; plots on the right show only exact matches. GPT-4 is the underlying model.
• Figure 3: Performance vs. memorization in different ICL regimes for all settings. Left plots show performance, and right plots display memorization across all settings. Memorization plots are duplicated from Figure \ref{['fig:memorization-quantification']} for comparison.
• Figure 5: An illustration of the few-shot ICL prompt used for classifying generated completions into exact, near-exact, or inexact matches using GPT-4. In this illustration, examples 1 through 4 form the fixed part of the input prompt, while example 5 is replaced with a new reference text (original subsequent segment of a dataset instance) and candidate text (LLM-generated completion) for evaluation. Example 1 is an exact match. Example 2 is a near-exact match where the candidate text has substantial overlap with the reference text but includes extra details. Examples 3 and 4 also show near-exact matches, where the candidate text is both semantically and structurally similar to the reference text.
• Figure : (1) Full Information