Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement

Joseph Shtok, Amit Alfassy, Foad Abo Dahood, Eliyahu Schwartz, Sivan Doveh, Assaf Arbelle

TL;DR

This work proposes Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the steps needed to solve a task using a few examples, starting from a small seed of manually crafted examples.

Abstract

It has been shown that Large Language Models' (LLMs) performance can be improved for many tasks using Chain of Thought (CoT) or In-Context Learning (ICL), which involve demonstrating the steps needed to solve a task using a few examples. However, while datasets with input-output pairs are relatively easy to produce, providing demonstrations which include intermediate steps requires cumbersome manual work. These steps may be executable programs, as in agentic flows, or step-by-step reasoning as in CoT. In this work, we propose Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the above intermediate steps, starting from a small seed of manually crafted examples. We demonstrate the advantage of ADLR in code-based table QA and mathematical reasoning, achieving up to a 5.5% gain. The code implementing our method is provided in the Supplementary material and will be made available.

Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement

TL;DR

This work proposes Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the steps needed to solve a task using a few examples, starting from a small seed of manually crafted examples.

Abstract

It has been shown that Large Language Models' (LLMs) performance can be improved for many tasks using Chain of Thought (CoT) or In-Context Learning (ICL), which involve demonstrating the steps needed to solve a task using a few examples. However, while datasets with input-output pairs are relatively easy to produce, providing demonstrations which include intermediate steps requires cumbersome manual work. These steps may be executable programs, as in agentic flows, or step-by-step reasoning as in CoT. In this work, we propose Automatic Data Labeling and Refinement (ADLR), a method to automatically generate and filter demonstrations which include the above intermediate steps, starting from a small seed of manually crafted examples. We demonstrate the advantage of ADLR in code-based table QA and mathematical reasoning, achieving up to a 5.5% gain. The code implementing our method is provided in the Supplementary material and will be made available.

Paper Structure

This paper contains 27 sections, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Prior Art
  3. Automatic prompt selection and generation for the In-Context Learning
  4. Table QA with code LLMs
  5. The Automatic Data Labeling and Refinement (ADLR) method
  6. Part I - generating a large pool of examples
  7. Part II - focusing on the useful examples
  8. Part III - augmenting the inference protocol
  9. Experiments
  10. Experimental setup
  11. Selecting an LLM
  12. ADLR parameters and inference protocols
  13. Finetuning an LLM
  14. Computational resources
  15. Results on Table QA task
  16. ...and 12 more sections

Figures (8)

  • Figure 1: From an input-output dataset with no intermediate steps (CoT/Executable programs), ADLR generates examples with such steps and retains the useful examples. These examples are used in the ICL to answer a given query.
  • Figure 2: Flow diagram of ADLR
  • Figure 3: Cumulative distribution of pool-A examples according to the sample difficulty metric $D(s)$
  • Figure 4: Histograms of success rate (percents of solved samples) for pool-B examples, applied in one-shot regime
  • Figure 5: Left: performance with different pools of examples. For WikiTQ dataset, we compare the vanilla and the finetuned CodeLlama-34b models (WikiTQ and WikiTQ-FT in the legend). Right: Performance as a number of shots for ADLR, compared to the baseline Binder (fixed context)
  • ...and 3 more figures