Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curate-Train-Refine: A Closed-Loop Agentic Framework for Zero Shot Classification

Gaurav Maheshwari, Kevin El Haddad

TL;DR

The paper tackles the practical challenge of deploying zero- and few-shot text classifiers when large LLMs are prohibitively costly at inference. It introduces a closed-loop, agentic framework in which an LLM acts as a data curator, generating seed data, training a lightweight classifier offline, diagnosing errors, and synthesizing targeted hard examples to iteratively refine supervision. Through a generate–evaluate–refine loop, the approach yields high-quality labeled data tailored to the downstream model, achieving strong zero- and few-shot performance across four benchmarks with SetFit and EuroBERT backbones. The results demonstrate that LLMs can serve as effective data curators, enabling accurate, low-latency classification without relying on large models at test time, and the work offers a pathway to more data-centric, efficient deployment of NLP systems.

Abstract

Large language models (LLMs) and high-capacity encoders have advanced zero and few-shot classification, but their inference cost and latency limit practical deployment. We propose training lightweight text classifiers using dynamically generated supervision from an LLM. Our method employs an iterative, agentic loop in which the LLM curates training data, analyzes model successes and failures, and synthesizes targeted examples to address observed errors. This closed-loop generation and evaluation process progressively improves data quality and adapts it to the downstream classifier and task. Across four widely used benchmarks, our approach consistently outperforms standard zero and few-shot baselines. These results indicate that LLMs can serve effectively as data curators, enabling accurate and efficient classification without the operational cost of large-model deployment.

Curate-Train-Refine: A Closed-Loop Agentic Framework for Zero Shot Classification

TL;DR

The paper tackles the practical challenge of deploying zero- and few-shot text classifiers when large LLMs are prohibitively costly at inference. It introduces a closed-loop, agentic framework in which an LLM acts as a data curator, generating seed data, training a lightweight classifier offline, diagnosing errors, and synthesizing targeted hard examples to iteratively refine supervision. Through a generate–evaluate–refine loop, the approach yields high-quality labeled data tailored to the downstream model, achieving strong zero- and few-shot performance across four benchmarks with SetFit and EuroBERT backbones. The results demonstrate that LLMs can serve as effective data curators, enabling accurate, low-latency classification without relying on large models at test time, and the work offers a pathway to more data-centric, efficient deployment of NLP systems.

Abstract

Large language models (LLMs) and high-capacity encoders have advanced zero and few-shot classification, but their inference cost and latency limit practical deployment. We propose training lightweight text classifiers using dynamically generated supervision from an LLM. Our method employs an iterative, agentic loop in which the LLM curates training data, analyzes model successes and failures, and synthesizes targeted examples to address observed errors. This closed-loop generation and evaluation process progressively improves data quality and adapts it to the downstream classifier and task. Across four widely used benchmarks, our approach consistently outperforms standard zero and few-shot baselines. These results indicate that LLMs can serve effectively as data curators, enabling accurate and efficient classification without the operational cost of large-model deployment.
Paper Structure (10 sections, 1 figure, 2 tables)

This paper contains 10 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Problem Setup
  4. Agentic Closed-Loop Framework
  5. Initialization.
  6. Iterative loop.
  7. Experiments
  8. Implementation Details
  9. Results
  10. Conclusion

Figures (1)

  • Figure 1: Overview of the proposed agentic training framework. (1) The LLM generates seed training data based on label semantics to train the initial classifier. (2) Performance metrics and specific error modes of the classifier are fed back to the LLM for analysis. (3) The LLM synthesizes targeted examples to address these deficiencies. This cycle repeats for a maximum of K iterations or until convergence, refining the classifier without human supervision.