Benchmarking General-Purpose In-Context Learning

TL;DR

The paper advocates General Purpose In-Context Learning (GPICL), extending ICL to long-horizon, diverse tasks and proposing two lightweight benchmarks, Meta-Language and Maze World, to study meta-training and ICL in this setting. Empirical results show that increasing task diversity improves GPICL generalization but can reduce zero-shot performance, while expanding the context length and memory states yields greater gains than merely scaling model size. The authors suggest a shift in learning pipelines toward meta-training followed by GPICL, leveraging synthetic data for efficiency and memory-rich architectures to handle long contexts. The work highlights significant potential for GPICL in language modeling and embodied navigation, while acknowledging the need for more realistic benchmarks and advanced memory-augmented models to realize practical benefits.

Abstract

In-context learning (ICL) empowers generative models to address new tasks effectively and efficiently on the fly, without relying on any artificially crafted optimization techniques. In this paper, we study extending ICL to address a broader range of tasks with an extended learning horizon and higher improvement potential, namely General Purpose In-Context Learning (GPICL). To this end, we introduce two lightweight benchmarks specifically crafted to train and evaluate GPICL functionalities. Each benchmark encompasses a vast number of tasks characterized by significant task variance. These tasks are also crafted to promote long-horizon in-context learning through continuous generation and interaction, covering domains such as language modeling, decision-making, and world modeling. The benchmarks necessitate the models to leverage contexts and history interactions to enhance their capabilities, which we believe to be the key characteristics of GPICL. Our experiments indicate that the diversity of training tasks is positively correlated with the ability to generalize with ICL, but inversely correlated with zero-shot capabilities. Additionally, our findings indicate that the scale of parameters alone may not be crucial for ICL or GPICL, suggesting alternative approaches such as increasing the scale of contexts and memory states.

Figures (19)

• Figure 1: Conceptual diagram of in-context learning (ICL) and general purpose in-context learning (GPICL)
• Figure 2: Learning pipelines based on GPICL versus the current LLM learning pipeline ouyang2022training.
• Figure 3: Comparing evaluations on meta-language sampled by different language complexity. Horizontal axis represent the position or context length, and vertical axis represent the perplexity (the lower the better).
• Figure 4: Comparing performances of language models of different sizes on the the same test set. Horizontal axis represent the position or context length, and vertical axis represent the perplexity (the lower the better).
• Figure 5: A demonstration of the maze world tasks alongside privileged agents with smart navigation policy. The privileged agent is provided with limited access to the environment's ground truth (b) under specific conditions. Only the areas within the agent's line of sight are revealed, while the unsighted regions remain obscured. The agent retains the visual information from the nearest three frames in its short-term memory, which is then transferred to long-term memory based on a specified probability. The privileged agent employs an exploration-then-exploitation strategy.