Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Submodular Context Partitioning and Compression for In-Context Learning

Shaoyi Zheng, Canyu Zhang, Tianyi Zhou, Shengjie Wang

TL;DR

Sub-CP introduces a submodular, block-aware context selection framework for in-context learning that controls block diversity from global coverage to local coherence. By employing the facility location function and four partitioning strategies, it enables precomputation and plug-and-play integration with ICAE, DENSE, and CEPE, yielding consistent improvements across five datasets and multiple model scales. The approach balances information coverage and redundancy, and analyses reveal robust gains on hard tasks while outlining areas for enhancement such as query-aware selection and explicit balance constraints. Overall, Sub-CP provides a principled, flexible method to structure exemplars for scalable, effective ICL in large language models.

Abstract

In-context learning (ICL) enables efficient few-shot learning in large language models (LLMs) without training, but suffers from the quadratic input complexity of transformers, limiting the maximum number of exemplars. While various efficient ICL approaches partition the context into blocks to process (e.g., ensembling, compression, cross-attention), they often ignore the information redundancy or under-representation caused by different partition strategies, leading to suboptimal performance. To tackle this problem, we propose Sub-CP, a block-aware context selection framework that leverages submodular objectives to control block diversity. Sub-CP supports a flexible spectrum of selection strategies, allowing each block to range from globally diverse to locally coherent. This allows fine-grained control over semantic structure while enabling precomputation. Extensive experiments across diverse tasks on multiple datasets show that Sub-CP consistently improves performance across model scales.

Submodular Context Partitioning and Compression for In-Context Learning

TL;DR

Sub-CP introduces a submodular, block-aware context selection framework for in-context learning that controls block diversity from global coverage to local coherence. By employing the facility location function and four partitioning strategies, it enables precomputation and plug-and-play integration with ICAE, DENSE, and CEPE, yielding consistent improvements across five datasets and multiple model scales. The approach balances information coverage and redundancy, and analyses reveal robust gains on hard tasks while outlining areas for enhancement such as query-aware selection and explicit balance constraints. Overall, Sub-CP provides a principled, flexible method to structure exemplars for scalable, effective ICL in large language models.

Abstract

In-context learning (ICL) enables efficient few-shot learning in large language models (LLMs) without training, but suffers from the quadratic input complexity of transformers, limiting the maximum number of exemplars. While various efficient ICL approaches partition the context into blocks to process (e.g., ensembling, compression, cross-attention), they often ignore the information redundancy or under-representation caused by different partition strategies, leading to suboptimal performance. To tackle this problem, we propose Sub-CP, a block-aware context selection framework that leverages submodular objectives to control block diversity. Sub-CP supports a flexible spectrum of selection strategies, allowing each block to range from globally diverse to locally coherent. This allows fine-grained control over semantic structure while enabling precomputation. Extensive experiments across diverse tasks on multiple datasets show that Sub-CP consistently improves performance across model scales.

Paper Structure

This paper contains 38 sections, 7 equations, 1 figure, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Global Diverse
  4. Global-Local Diverse
  5. Local Diverse
  6. Local Coherent
  7. Experiment Result
  8. Dataset
  9. Experiment
  10. Demonstration Ensembling (DENSE)
  11. Baseline
  12. Qualitative Comparison
  13. In-context Autoencoder (ICAE)
  14. Baseline
  15. Qualitative Comparison
  16. ...and 23 more sections

Figures (1)

  • Figure 1: The Sub-CP overview. Sub-CP selects and partitions from the whole dataset and then distributes them into blocks in four ways, ranging from global diverse to local coherent.