Language Models in the Loop: Incorporating Prompting into Weak Supervision

TL;DR

The paper addresses the challenge of limited labeled data by integrating large language model prompting into a programmatic weak supervision framework. By treating prompting outputs as labeling functions and denoising their signals with a label model, the authors train end classifiers that outperform zero-shot baselines and rival hand-engineered labeling functions. They demonstrate significant accuracy gains on the WRENCH benchmark, with especially strong results for the T0++ model and benefits from contextual calibration and label-function diversity. This approach offers a practical path to harness foundation-model knowledge for task-specific supervision without extensive hand-coding or fine-tuning. The work has practical implications for rapidly building accurate classifiers in data-scarce domains using accessible LM prompting and weak supervision pipelines.

Abstract

We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct queries about an example and define how the possible responses should be mapped to votes for labels and abstentions. We then denoise these noisy label sources using the Snorkel system and train an end classifier with the resulting training data. Our experimental evaluation shows that prompting large language models within a weak supervision framework can provide significant gains in accuracy. On the WRENCH weak supervision benchmark, this approach can significantly improve over zero-shot performance, an average 19.5% reduction in errors. We also find that this approach produces classifiers with comparable or superior accuracy to those trained from hand-engineered rules.

Figures (13)

• Figure 1: An overview of how a subject matter expert (SME) can use prompting to create weak supervision sources. The SME expresses tests for signifiers of the class of interest as natural language prompts. The prompts are combined with unlabeled examples and given to a pre-trained language model. The model's responses are mapped to votes on the true label for the example.
• Figure 2: Language models in the loop: the overall framework for developing and applying prompted labeling functions. The subject matter expert (SME) expresses their domain knowledge via prompts that are combined with unlabeled examples and given to a pre-trained language model. The model's responses are interpreted with label maps to produce votes on the true label. These votes are denoised with a label model, and the resulting estimated labels are used to train an end model. Throughout the process, the SME can refine their prompts by inspecting unlabeled examples and evaluating with a small labeled development set.
• Figure 3: A comparison of a regular expression (RegEx) labeling function from the WRENCH benchmark zhang:neurips21 with the corresponding prompted labeling function using the T0++sanh:iclr22 pre-trained language model (PLM). The regular expression looks for variations of the phrase "check out" and the prompted labeling function uses the prompt "Does the following comment contain the words 'check out'?" RegEx has 100% precision and 45% recall, while PLM has 76% precision and 58% recall. This comparison shows that even simple labeling functions can be made more general while maintaining acceptable precision by using prompting.
• Figure 4: SMS prompted labeling function coverage (x-axis) vs. accuracy (y-axis). The top figure is calibrated using contextual calibration and the bottom is uncalibrated. WRENCH Benchmark labeling function performance is in blue in every subfigure, which in SMS favors high precision, extremely low-coverage ($<2\%$).
• Figure 5: Absolute change in accuracy and coverage after contextual calibration for all prompted labeling functions and language models. Each subfigure contains points from all datasets. The x-axis is change in coverage, the y-axis is change in accuracy, and each point reflects the change in that prompt's labeling performance after calibration.