Improving Classification Performance With Human Feedback: Label a few, we label the rest
Natan Vidra, Thomas Clifford, Katherine Jijo, Eden Chung, Liang Zhang
TL;DR
The paper tackles labeled-data scarcity for text classification by combining few-shot learning with a continuous human-in-the-loop feedback process. It leverages LLMs such as GPT-3.5 Turbo, BERT, and SetFit to make initial predictions with zero-shot baselines, then incrementally correct 10 erroneous cases per cycle and retrain, facilitating entropy-guided edge-case labeling. Across six datasets (Amazon, Banking, Craigslist, Financial Phrasebank, TREC), the approach yields consistent improvements in accuracy, precision, and recall, often surpassing zero-shot baselines, with results and code available on GitHub. The work demonstrates a practical pathway to high-performing domain-specific classifiers with far fewer labeled examples, reducing labeling costs and enabling rapid adaptation to changing data requirements.
Abstract
In the realm of artificial intelligence, where a vast majority of data is unstructured, obtaining substantial amounts of labeled data to train supervised machine learning models poses a significant challenge. To address this, we delve into few-shot and active learning, where are goal is to improve AI models with human feedback on a few labeled examples. This paper focuses on understanding how a continuous feedback loop can refine models, thereby enhancing their accuracy, recall, and precision through incremental human input. By employing Large Language Models (LLMs) such as GPT-3.5, BERT, and SetFit, we aim to analyze the efficacy of using a limited number of labeled examples to substantially improve model accuracy. We benchmark this approach on the Financial Phrasebank, Banking, Craigslist, Trec, Amazon Reviews datasets to prove that with just a few labeled examples, we are able to surpass the accuracy of zero shot large language models to provide enhanced text classification performance. We demonstrate that rather than needing to manually label millions of rows of data, we just need to label a few and the model can effectively predict the rest.