Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting

Longyu Feng, Mengze Hong, Chen Jason Zhang

TL;DR

The proposed Auto-Demo Prompting effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage and opening new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.

Abstract

Batch prompting is a common technique in large language models (LLMs) used to process multiple inputs simultaneously, aiming to improve computational efficiency. However, as batch sizes increase, performance degradation often occurs due to the model's difficulty in handling lengthy context inputs. Existing methods that attempt to mitigate these issues rely solely on batch data arrangement and majority voting rather than improving the design of the batch prompt itself. In this paper, we address these limitations by proposing "Auto-Demo Prompting," a novel approach that leverages the question-output pairs from earlier questions within a batch as demonstrations for subsequent answer inference. We provide a formal theoretical analysis of how Auto-Demo Prompting functions within the autoregressive generation process of LLMs, illustrating how it utilizes prior outputs to optimize the model's internal representations. Our method effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage. Experimental results across five NLP tasks demonstrate its effectiveness in mitigating performance degradation and occasionally outperforming single prompts. Furthermore, it opens new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.

Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting

TL;DR

The proposed Auto-Demo Prompting effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage and opening new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.

Abstract

Batch prompting is a common technique in large language models (LLMs) used to process multiple inputs simultaneously, aiming to improve computational efficiency. However, as batch sizes increase, performance degradation often occurs due to the model's difficulty in handling lengthy context inputs. Existing methods that attempt to mitigate these issues rely solely on batch data arrangement and majority voting rather than improving the design of the batch prompt itself. In this paper, we address these limitations by proposing "Auto-Demo Prompting," a novel approach that leverages the question-output pairs from earlier questions within a batch as demonstrations for subsequent answer inference. We provide a formal theoretical analysis of how Auto-Demo Prompting functions within the autoregressive generation process of LLMs, illustrating how it utilizes prior outputs to optimize the model's internal representations. Our method effectively bridges the gap between batch prompting and few-shot prompting, enhancing performance with only a slight compromise in token usage. Experimental results across five NLP tasks demonstrate its effectiveness in mitigating performance degradation and occasionally outperforming single prompts. Furthermore, it opens new avenues for applying few-shot learning techniques, such as demonstration selection, within batch prompting, making it a robust solution for real-world applications.
Paper Structure (18 sections, 2 equations, 4 figures, 6 tables, 4 algorithms)

This paper contains 18 sections, 2 equations, 4 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Auto-Demo Prompting
  3. Autoregressive Generation Process in Large Language Model
  4. Autoregressive Generation of Demonstrations
  5. Batch Data Selection
  6. Experiments
  7. Experimental Settings
  8. Datasets
  9. Implementations
  10. Results and Discussions
  11. Batch Data Selection Experiments
  12. Discussion
  13. Related Work
  14. Batch Prompting
  15. Demonstration Selection in In-Context Learning
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Example: a) Batch Prompting and b) Auto-Demo Prompting
  • Figure 2: Auto-Demo Prompting: Single prompts are combined into a batch prompt with a special output control for generating question-answer pairs, along with optional batch data selection. This prompt is fed into the autoregressive generation process of a decoder-only LLM, forming demonstrations for subsequent generation.
  • Figure 3: Experimental results: accuracy for Auto-Demo Prompt ("w") vs. Batch Prompt ("w/o") across different models and batch sizes.
  • Figure 4: Experimental results for Batch Data Selection: Auto-Demo Prompting ("w") vs. Batch Prompting ("w/o"); "+bds" signifies the application of batch data selection with retrieval.