Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Text Classification with Conformal In-Context Learning

Ippokratis Pantelidis, Korbinian Randl, Aron Henriksson

TL;DR

CICLe combines a lightweight base classifier with Conformal Prediction to adaptively narrow the candidate label set for LLM prompting, reducing context length and computational cost. The paper presents a systematic cross-domain evaluation across four datasets, showing consistent performance gains over the base classifier and competitive results with few-shot prompting, especially in imbalanced and data-rich regimes. It also demonstrates substantial efficiency: fewer shots and shorter prompts, enabling smaller LLMs to achieve competitive results, with particular strength on highly imbalanced tasks. The work provides practical guidance on when CICLe is advantageous and highlights directions for extending conformal-in-context learning to broader NLP tasks.

Abstract

Large Language Models (LLMs) demonstrate strong in-context learning abilities, yet their effectiveness in text classification depends heavily on prompt design and incurs substantial computational cost. Conformal In-Context Learning (CICLe) has been proposed as a resource-efficient framework that integrates a lightweight base classifier with Conformal Prediction to guide LLM prompting by adaptively reducing the set of candidate classes. However, its broader applicability and efficiency benefits beyond a single domain have not yet been systematically explored. In this paper, we present a comprehensive evaluation of CICLe across diverse NLP classification benchmarks. The results show that CICLe consistently improves over its base classifier and outperforms few-shot prompting baselines when the sample size is sufficient for training the base classifier, and performs comparably in low-data regimes. In terms of efficiency, CICLe reduces the number of shots and prompt length by up to 34.45% and 25.16%, respectively, and enables the use of smaller models with competitive performance. CICLe is furthermore particularly advantageous for text classification tasks with high class imbalance. These findings highlight CICLe as a practical and scalable approach for efficient text classification, combining the robustness of traditional classifiers with the adaptability of LLMs, and achieving substantial gains in data and computational efficiency.

Efficient Text Classification with Conformal In-Context Learning

TL;DR

CICLe combines a lightweight base classifier with Conformal Prediction to adaptively narrow the candidate label set for LLM prompting, reducing context length and computational cost. The paper presents a systematic cross-domain evaluation across four datasets, showing consistent performance gains over the base classifier and competitive results with few-shot prompting, especially in imbalanced and data-rich regimes. It also demonstrates substantial efficiency: fewer shots and shorter prompts, enabling smaller LLMs to achieve competitive results, with particular strength on highly imbalanced tasks. The work provides practical guidance on when CICLe is advantageous and highlights directions for extending conformal-in-context learning to broader NLP tasks.

Abstract

Large Language Models (LLMs) demonstrate strong in-context learning abilities, yet their effectiveness in text classification depends heavily on prompt design and incurs substantial computational cost. Conformal In-Context Learning (CICLe) has been proposed as a resource-efficient framework that integrates a lightweight base classifier with Conformal Prediction to guide LLM prompting by adaptively reducing the set of candidate classes. However, its broader applicability and efficiency benefits beyond a single domain have not yet been systematically explored. In this paper, we present a comprehensive evaluation of CICLe across diverse NLP classification benchmarks. The results show that CICLe consistently improves over its base classifier and outperforms few-shot prompting baselines when the sample size is sufficient for training the base classifier, and performs comparably in low-data regimes. In terms of efficiency, CICLe reduces the number of shots and prompt length by up to 34.45% and 25.16%, respectively, and enables the use of smaller models with competitive performance. CICLe is furthermore particularly advantageous for text classification tasks with high class imbalance. These findings highlight CICLe as a practical and scalable approach for efficient text classification, combining the robustness of traditional classifiers with the adaptability of LLMs, and achieving substantial gains in data and computational efficiency.

Paper Structure

This paper contains 20 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Base Classifier
  5. Few-Shot Prompting
  6. CICLe Framework
  7. Experimental Setup
  8. Datasets
  9. Base Classifier
  10. Large Language Models
  11. Prompt Construction
  12. Implementation Details
  13. Results
  14. Predictive Performance
  15. Resource Efficiency
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Macro F$_1$ vs number of samples in the training set. CICLe performs comparably to other strategies in balanced datasets and outperforms its competition in the imbalanced scenario.
  • Figure 2: Prompt size vs. number of samples in the training set for CICLe vs. few-shot baselines (N.B. model size and shot selection strategy have no impact). CICLe consistently uses a lower number of shots in its prompts, with this number decreasing as the number of training samples increases and the base classifier gets better.