Are Human-generated Demonstrations Necessary for In-context Learning?

TL;DR

This paper investigates whether human-generated demonstrations are necessary for in-context learning (ICL) and introduces Self-Contemplation Prompting (SEC), a paradigm in which the LLM itself creates demonstrations before producing outputs. SEC has two variants: Vanilla SEC, which generates input–label demonstrations, and CoT-SEC, which additionally generates the reasoning steps, enabling chain-of-thought-style answers without human-crafted rationales. Across arithmetic reasoning, commonsense reasoning, multi-task language understanding, and code generation, SEC consistently beats zero-shot prompts and achieves performance comparable to or surpassing ICL with hand-crafted demonstrations, illustrating that contemporary LLMs can rely on their own capacity to generate illustrative examples. The results suggest that annotated training data may be dispensable for many tasks, and SEC provides a flexible, resource-efficient framework with broad applicability; code is released at the provided GitHub repository.

Abstract

Despite the promising few-shot ability of large language models (LLMs), the standard paradigm of In-context Learning (ICL) suffers the disadvantages of susceptibility to selected demonstrations and the intricacy to generate these demonstrations. In this paper, we raise the fundamental question that whether human-generated demonstrations are necessary for ICL. To answer this question, we propose self-contemplation prompting strategy (SEC), a paradigm free from human-crafted demonstrations. The key point of SEC is that, instead of using hand-crafted examples as demonstrations in ICL, SEC asks LLMs to first create demonstrations on their own, based on which the final output is generated. SEC is a flexible framework and can be adapted to both the vanilla ICL and the chain-of-thought (CoT), but with greater ease: as the manual-generation process of both examples and rationale can be saved. Extensive experiments in arithmetic reasoning, commonsense reasoning, multi-task language understanding, and code generation benchmarks, show that SEC, which does not require hand-crafted demonstrations, significantly outperforms the zero-shot learning strategy, and achieves comparable results to ICL with hand-crafted demonstrations. This demonstrates that, for many tasks, contemporary LLMs possess a sufficient level of competence to exclusively depend on their own capacity for decision making, removing the need for external training data. Code is available at https://github.com/ruili33/SEC.

Figures (20)

• Figure 1: Comparison between vanilla ICL and vanilla SEC. Different parts of the prompt and results are highlighted with different colors for emphasis.
• Figure 2: Comparison between CoT-ICL and CoT-SEC. Different parts of the prompt and results are highlighted with different colors for emphasis.
• Figure 3: Experiment results on the MATH dataset by subtopic.
• Figure 4: Effects of the number of examples on the GSM8K and HumanEval datasets. Solid data points represent the data points adopted in the main results.
• Figure 5: The performance of all four prompting strategies on three models in the GPT3.5 family on GSM8K. SEC is an emergent ability.