Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

Dingzirui Wang, Xuanliang Zhang, Qiguang Chen, Longxu Dou, Xiao Xu, Rongyu Cao, Yingwei Ma, Qingfu Zhu, Wanxiang Che, Binhua Li, Fei Huang, Yongbin Li

TL;DR

In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks by defining an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task.

Abstract

In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.

In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

TL;DR

In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks by defining an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task.

Abstract

In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.
Paper Structure (46 sections, 1 theorem, 12 equations, 6 figures, 13 tables)

This paper contains 46 sections, 1 theorem, 12 equations, 6 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Demonstration Synthesis
  4. Deep Transfer Learning
  5. Methodology
  6. Source Sampling
  7. Optimization Objective for Source Sample
  8. Sampling with Equation \ref{['equ:source_dataset_optimization']}
  9. Target Transfer
  10. Transfer
  11. Verify
  12. Sample
  13. Experiments
  14. Experiment Setup
  15. Dataset
  16. ...and 31 more sections

Key Result

Theorem 1

Let $x_S, x_T$ represent the representation vectors of the task definition of $S$ and $T$. If then

Figures (6)

  • Figure 1: Comparison between previous demonstration synthesis methods (top) and our method (bottom). The blue part denotes the definition of the target task. The previous method synthesizes demonstration from scratch, while the model misinterprets the definition and generates a demonstration with the wrong answer, where the answer is not explicit mentioned by the sentence. In contrast, our method synthesizes demonstrations by transferring the sampled demonstrations, reducing the reliance on the capabilities of LLMs. The corresponding parts between the source and the target demonstrations of our method are marked in bold.
  • Figure 2: The illustration of ICTL, taking the target task definition "If the provided sentence contains an explicit mention that answers the given question" as an example. ICTL consists of two steps: (i) Source Sampling: sample demonstrations that are similar to the target task from the source tasks; (ii) Target Transfer: transfer the sampled demonstrations to the target task. The blue part indicates the task definitions and demonstrations similar to the target task, and the gray part indicates that it is dissimilar. The green part denotes the transferred demonstrations.
  • Figure 3: The impact of different parameters on the performance of the Super-NI test set with ICTL using Llama3.1-8b. $0$ of the X-axis indicates the performance under the Single setting.
  • Figure 4: Category distribution of the Super-NI test set.
  • Figure 5: Rouge on the Super-NI test set using the 32 different sets of randomly sampled transferred demonstrations with different values of Equation \ref{['equ:target_dataset_optimization']} using Llama3.1-8b. To better observe the changes, we normalize the values of the X-axis.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Proof 1