StaICC: Standardized Evaluation for Classification Task in In-context Learning

TL;DR

The paper tackles the problem of inconsistent ICC (in-context classification) results caused by varying prompts, sampling, and demonstrations. It introduces StaICC, a standardized evaluation toolkit, including StaICC-Normal for a fixed, dataset-backed benchmark and StaICC-Diag for diagnostic analyses of bias and robustness, validated across 29 modern LMs and 10 inference methods. The key contributions are a fixed data-processing pipeline, a 10-dataset standardized ICC benchmark with four metrics, and a diagnostic suite that probes prediction bias and robustness, enabling fair cross-study comparisons and more reliable meta-analyses. This standardization enhances reproducibility and provides actionable baselines for improving ICL methods, with demonstrated scaling laws and detailed diagnostic insights guiding future research.

Abstract

Classification tasks are widely investigated in the In-Context Learning (ICL) paradigm. However, current efforts are evaluated on disjoint benchmarks and settings, while their performances are significantly influenced by some trivial variables, such as prompt templates, data sampling, instructions, etc., which leads to significant inconsistencies in the results reported across various literature, preventing fair comparison or meta-analysis across different papers. Therefore, this paper proposes a standardized and easy-to-use evaluation toolkit (StaICC) for in-context classification. Including, for the normal classification task, we provide StaICC-Normal, selecting 10 widely used datasets, and generating prompts with a fixed form, to mitigate the variance among the experiment implementations. To enrich the usage of our benchmark, we also provide a sub-benchmark StaICC-Diag for diagnosing ICL from several aspects, aiming for a more robust inference processing.

Figures (8)

• Figure 1: Summaries of experimental results in the literature: ICL results are disjoint in the literature. Left: Accuracies of various models with different sizes on specified datasets, where the results do not comply with the scaling laws w.r.t. the model scaling. Right: Accuracies of the same model (GPT2-XL) on vanilla ICL inference from different papers, where results have a considerable range even on the same model.
• Figure 2: The influence of various prompt variables on ICL. Details are shown in Appendix \ref{['appendix:exp_detail']}.
• Figure 3: Major schematic diagram of data pre-processing and input forming in StaICC. Raw datasets are first divided into 3 sub-sets with a frozen pre-processer for calibration, demonstration, and query. Based on these sub-sets, ICL inputs are built with a frozen demonstration sampler, and a frozen prompt template under a meta-template, shown in gray for an example.
• Figure 4: TLP results against model parameter numbers.
• Figure 5: Absolute accuracy (%) improvement of various ICL-improving methods against model parameter numbers.