Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Yong Huang, Wei Lu

TL;DR

<3-5 sentence high-level summary> This paper tackles how the order of in-context demonstrations affects in-context learning performance in LLMs. It introduces In-Context Curriculum Learning (ICCL), a straightforward method that arranges demonstrations from easy to hard, using perplexity as a proxy for difficulty, and validates it at both corpus and instance levels on open-source LLMs. The study shows ICCL yields stable, significant gains over baselines across multiple scientific NLP tasks and models, and reveals that the ICCL capability largely emerges during instruction-tuning rather than pretraining. Code release facilitates replication and further exploration of curriculum-based prompting in ICL.

Abstract

Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require high computational costs to introduce the priori knowledge. In this paper, inspired by the human learning process, we propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL). The ICCL implies gradually increasing the complexity of prompt demonstrations during the inference process. The difficulty can be assessed by human experts or LLMs-driven metrics, such as perplexity. Then we design extensive experiments to discuss the effectiveness of the ICCL at both corpus-level and instance-level. Moreover, we also investigate the formation mechanism of LLM's ICCL capability. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for representative open-source LLMs. To facilitate further research and applications by other scholars, we make the code publicly available.

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning

TL;DR

<3-5 sentence high-level summary> This paper tackles how the order of in-context demonstrations affects in-context learning performance in LLMs. It introduces In-Context Curriculum Learning (ICCL), a straightforward method that arranges demonstrations from easy to hard, using perplexity as a proxy for difficulty, and validates it at both corpus and instance levels on open-source LLMs. The study shows ICCL yields stable, significant gains over baselines across multiple scientific NLP tasks and models, and reveals that the ICCL capability largely emerges during instruction-tuning rather than pretraining. Code release facilitates replication and further exploration of curriculum-based prompting in ICL.

Abstract

Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require high computational costs to introduce the priori knowledge. In this paper, inspired by the human learning process, we propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL). The ICCL implies gradually increasing the complexity of prompt demonstrations during the inference process. The difficulty can be assessed by human experts or LLMs-driven metrics, such as perplexity. Then we design extensive experiments to discuss the effectiveness of the ICCL at both corpus-level and instance-level. Moreover, we also investigate the formation mechanism of LLM's ICCL capability. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for representative open-source LLMs. To facilitate further research and applications by other scholars, we make the code publicly available.
Paper Structure (18 sections, 2 equations, 2 figures, 5 tables)

This paper contains 18 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Demonstrations Organization
  4. Curriculum Learning
  5. Methodology
  6. Problem Formulation
  7. Curriculum Schedule Construction
  8. Experiments
  9. Setup
  10. Datasets
  11. Models
  12. Baseline
  13. Main Result
  14. Formation Mechanism of ICCL Capability
  15. Conclusion
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Illustration of In-Context Curriculum Learning (ICCL). The curriculum schedule can be designed by both human and LLMs, schedule constructor sort demonstrations from easy to hard based on their understanding.
  • Figure 2: $F_1$ scores improvement (or decline) rates for both Based LLMs and Instruction-Tuned LLMs using ICCL compared with random order.