Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning

TL;DR

This study investigates how in-context learning (ICL) emerges by examining the pre-training dynamics of two underlying abilities: task recognition (TR) and task learning (TL). It reveals a widespread competitive relationship between TR and TL during pre-training, with a higher overall competition correlating negatively with final ICL performance. By analyzing model size, dataset size, and data curriculum, the authors identify practical levers to modulate this competition and enhance ICL. They further propose an adaptive ensemble method that fuses TR- and TL-strong checkpoints at inference, yielding significant ICL gains and enabling smaller models to surpass larger ones. The work offers new insights into ICL mechanisms and a viable, scalable approach to improve ICL via checkpoint fusion.

Abstract

The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demonstrations. However, relationships between the two abilities and how such relationships affect the emergence of ICL is unclear. In this paper, we take the first step by examining the pre-training dynamics of the emergence of ICL. With carefully designed metrics, we find that these two abilities are, in fact, competitive during pre-training. Moreover, we observe a strong negative correlation between the competition and ICL performance. Further analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) demonstrates possible ways to manage the competition. Based on these insights, we propose a simple yet effective method to better integrate these two abilities for ICL at inference time. Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters. The code is available at https://github.com/RUCAIBox/Competitive-ICL.

Figures (8)

• Figure 1: The performance of MiniCPM-2B and Amber-7B for ICL and its two abilities (i.e., task recognition and task learning). The emergence of ICL encounters many fluctuations, where the performance of task recognition and task learning changes in the opposite direction.
• Figure 2: Average ratio of competition for LLMs.
• Figure 3: The performance of ICL and the evolution of competition ($R_i$) during the pre-training of MiniCPM-2B and Amber-7B.
• Figure 4: ICL performance of the final checkpoint and the average intensity of competition ($\bar{C}^s$) for LLMs.
• Figure 5: The evolution and average intensity ($\bar{C_i^s}$) of competition of LLMs with different model sizes.