Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions

TL;DR

This work investigates why in-context learning fails or underperforms in certain settings by examining how context usefulness depends on question type, difficulty, and novelty. It introduces a novel benchmark of 160 open-ended physics and computer science questions with four context types and uses GPT-4 responses graded by multiple experts to quantify effects across contexts. The study contrasts open-ended results with existing closed-form benchmarks (MetaICL, NephSAP) and replicates prior findings, revealing that context relevance often helps closed-form tasks but can hurt open-ended ones, with difficulty and originality further modulating these effects. The findings have practical implications for retrieval-augmented generation systems, suggesting that context selection should be tailored to question form and application, and that shell-based or diversity-enhancing retrieval strategies may be more robust than proximity-based approaches.

Abstract

We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each paired with a context of various relevancy. We show that counter-intuitively, a context that is more aligned with the topic does not always help more than a less relevant context. This effect is especially visible for open questions and questions of high difficulty or novelty. This result reveals a fundamental difference between the treatment of close-form and open-form questions by large-language models and shows a need for a more robust evaluation of in-context learning on the variety of different types of questions. It also poses a new question of how to optimally select a context for large language models, especially in the context of Retrieval Augmented Generation (RAG) systems. Our results suggest that the answer to this question can be highly application-dependent and might be contingent on factors including the format of the question, the perceived difficulty level of the questions, and the novelty or popularity of the information we seek.

Figures (8)

• Figure 1: (A) Raw average scores of generated responses for each context type (no context, irrelevant context, vague context, and relevant context) evaluated for Completeness and Relevancy (Correctness), Logic and Reasoning (Logic Score), and Truthfulness (lack of hallucination), assessed by six different graders. (B) The process of standardizing raw scores from each grader to calculate the overall standardized average scores. The raw scores are converted to Z-scores, which are then averaged to obtain standardized average scores. (C) Standardized average scores of generated responses for each context type aggregated across all graders.
• Figure 2: (A): Standardized average scores of generated responses for each context type (no context, irrelevant context, vague context, and relevant context), categorized by three levels of question difficulty (easy, medium, and hard) for correctness, logic errors, and lack of hallucination. (B): Standardized average scores of generated answers for each context type, subdivided into known, paraphrased, and original categories, evaluated for correctness, logic score, and lack of hallucination.
• Figure 3: Comparison of results for different datasets. (a) Results for the MetalCL dataset. (b) Results for the NephSAP dataset. (c) Results for the Open dataset.
• Figure 4: (A) A typical hypershpere, from which we sample documents in RAG applications. (B) An alternative approach, where we either exclude or at least diminish the impact of contexts that are too close to the point representing the querry.
• Figure 5: The potato grading interface used in evaluation