Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving In-Context Learning with Small Language Model Ensembles

M. Mehdi Mojarradi, Lingyi Yang, Robert McCraith, Adam Mahdi

TL;DR

This work tackles the gap between cheap in-context learning and costly fine-tuning by introducing Ensemble SuperICL, which leverages predictions and confidence scores from multiple small language models within the ICL prompt. By concatenating SLM-derived demonstrations and confidences with test inputs, an LLM can better triangulate correct labels without dataset-specific fine-tuning. The approach achieves SoTA performance on several GLUE benchmarks and demonstrates practical gains on a medical labeling task (MedMCQA), with comprehensive ablation and sensitivity analyses confirming the contribution of each component. The method shows promise for efficient domain specialization in real-world settings where resources are constrained.

Abstract

Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.

Improving In-Context Learning with Small Language Model Ensembles

TL;DR

This work tackles the gap between cheap in-context learning and costly fine-tuning by introducing Ensemble SuperICL, which leverages predictions and confidence scores from multiple small language models within the ICL prompt. By concatenating SLM-derived demonstrations and confidences with test inputs, an LLM can better triangulate correct labels without dataset-specific fine-tuning. The approach achieves SoTA performance on several GLUE benchmarks and demonstrates practical gains on a medical labeling task (MedMCQA), with comprehensive ablation and sensitivity analyses confirming the contribution of each component. The method shows promise for efficient domain specialization in real-world settings where resources are constrained.

Abstract

Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.

Paper Structure

This paper contains 15 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Ensemble Super In-Context Learning
  3. Methods
  4. Results and Discussion
  5. Ensemble SuperICL outperforms nearly all baselines on each dataset
  6. Ablation study: all components of Ensemble SuperICL are significant
  7. Limitations and future works
  8. Conclusion
  9. Datasets
  10. Example of Ensemble SuperICL procedure on the SST-2 dataset
  11. Prompt designs for each dataset
  12. Full results of experiments on benchmark datasets
  13. Full results of experiments from MedMCQA case study
  14. Best hyperparameters of Ensemble SuperICL
  15. Variance in Ensemble SuperICL is generally low

Figures (2)

  • Figure 1: Three stages in Ensemble SuperICL.In-context examples: $k$ input-true label pairs are sampled from a dataset, along with a test input. Ensemble super context: the examples are passed to $p$ small language models (SLMs) which each produce predicted labels and confidence scores, for the examples and the test input. Inference: a large language model (LLM) is given an instruction prompt along with the SLM-enhanced examples and test input, and must predict the label of the test input.
  • Figure 2: Example of Ensemble SuperICL procedure on the SST-2 dataset.