Incubating Text Classifiers Following User Instruction with Nothing but LLM

TL;DR

This work tackles the problem of building text classifiers from user instructions without requiring human-annotated corpora by introducing Incubator, an instruction-tuned LLM framework. It learns instruction-to-data mappings from HuggingFace descriptions and uses GPT-4 for in-context augmentation, followed by self-diversification via clustering to enhance data uniformity and semantic coverage. The framework supports complex, mutually dependent labels and enables logical text mining by incubating multiple classifiers, achieving strong results on traditional benchmarks and in personalized/miscellaneous settings. The approach demonstrates practical potential for rapid, user-driven classifier development with efficient inference using open-source models, while outlining limitations related to instruction effort and knowledge scope and proposing avenues for future improvement.

Abstract

In this paper, we aim to generate text classification data given arbitrary class definitions (i.e., user instruction), so one can train a small text classifier without any human annotation or raw corpus. Compared with pioneer attempts, our proposed Incubator is the first framework that can handle complicated and even mutually dependent classes (e.g., "TED Talk given by Educator" and "Other"). Specifically, Incubator is an LLM firstly tuned on the instruction-to-data mappings that we obtained from classification datasets and descriptions on HuggingFace together with in-context augmentation by GPT-4. We then refine Incubator by learning on the cluster centers of semantic textual embeddings to emphasize the uniformity and semantic diversity in generations. We compare Incubator on various classification tasks with strong baselines such as direct LLM-based inference and training data generation by prompt engineering. Experiments show Incubator is able to (1) perform well on traditional benchmarks, (2) take label dependency and user preference into consideration, and (3) enable logical text mining by incubating multiple classifiers.

Figures (6)

• Figure 1: A comparison of Incubator with different methods for zero-shot text classification.
• Figure 2: An overview of our framework to build Incubator.
• Figure 3: Incubation dataset size analysis.
• Figure 4: Analysis of Incubator instruction robustness.
• Figure 5: Efficiency Analysis of Incubator instruction.