Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions

Yiming Tang, Bin Dong

TL;DR

A novel prompt engineering workflow built around a novel object called the "demonstrative regime" of a demonstration is proposed, providing valuable insights into how demonstrations relate to different question types within a dataset.

Abstract

Large language models (LLMs) benefit greatly from prompt engineering, with in-context learning standing as a pivital technique. While former approaches have provided various ways to construct the demonstrations used for in-context learning, they often ignore the inherent heterogeneity within datasets, applying the same demonstrations to all reasoning questions. We observed that the effectiveness of demonstrations varies depending on the specific question. This motivates our exploration of using prompt engineering to select appropriate demonstrations. To address the challenge of automatically creating and choosing demonstrations tailored to each question, we propose a novel prompt engineering workflow built around a novel object called the "demonstration notebook." This notebook helps identify the most suitable in-context learning example for a question by gathering and reusing information from the LLM's past interactions. Our experiments show that this approach outperforms all existing methods for automatic demonstration construction and selection (as far as we know), achieving state-of-the-art results on serveral reasoning benchmarks. The method's versatility is further demonstrated by its success in text summarization and prompt compression tasks. Additionally, we contribute a rigorous analysis method to reveal the "demonstrative regime" of a demonstration, providing valuable insights into how demonstrations relate to different question types within a dataset.

Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions

TL;DR

A novel prompt engineering workflow built around a novel object called the "demonstrative regime" of a demonstration is proposed, providing valuable insights into how demonstrations relate to different question types within a dataset.

Abstract

Large language models (LLMs) benefit greatly from prompt engineering, with in-context learning standing as a pivital technique. While former approaches have provided various ways to construct the demonstrations used for in-context learning, they often ignore the inherent heterogeneity within datasets, applying the same demonstrations to all reasoning questions. We observed that the effectiveness of demonstrations varies depending on the specific question. This motivates our exploration of using prompt engineering to select appropriate demonstrations. To address the challenge of automatically creating and choosing demonstrations tailored to each question, we propose a novel prompt engineering workflow built around a novel object called the "demonstration notebook." This notebook helps identify the most suitable in-context learning example for a question by gathering and reusing information from the LLM's past interactions. Our experiments show that this approach outperforms all existing methods for automatic demonstration construction and selection (as far as we know), achieving state-of-the-art results on serveral reasoning benchmarks. The method's versatility is further demonstrated by its success in text summarization and prompt compression tasks. Additionally, we contribute a rigorous analysis method to reveal the "demonstrative regime" of a demonstration, providing valuable insights into how demonstrations relate to different question types within a dataset.
Paper Structure (15 sections, 2 figures, 3 tables, 3 algorithms)

This paper contains 15 sections, 2 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Overview
  5. Demonstration Expansion
  6. On-policy Collection
  7. Off-policy Collection
  8. Pruning
  9. Training the Prompter and Inference
  10. Experiments
  11. In-context Learning for Reasoning
  12. Main Results
  13. In-context Learning for Article Summarization
  14. Further Analysis on the Demonstrative Regime
  15. Conclusion

Figures (2)

  • Figure 1: A. The demonstration notebook consists of three components, the interactive record set, the demonstration set and the noted question set. The records in the interactive record set are in the form of (question demonstration) pairs indicating the demonstration is demonstrative to the question. B. The interactive records in the set are used to train a prompter for demonstration selection. We add an adapter module to a pretrained embedding model for this prompter. C. The collection phase of the demonstration notebook consists of several epochs each of four procedures. D. During the inference phase, a question is fed to the prompter for a selected demostration which is subsequently concatenated with the question when fed to LLMs as in-context examples.
  • Figure 2: Visualization of questions that can be solved by the LLM directly, and using three different demonstrations respectively. The green dots means the question that can be correctly solved and ther other dots refer to clusters of questions that can be solved with the demonstrations. The last two diagrams refer to the demonstrative regiems of three demonstrations for two datasets, SVAMP and MutlitArith.