Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks

TL;DR

This work introduces BoxWRENCH, a realistic weak supervision benchmark suite for text classification that emphasizes high class cardinality, imbalance, and domain-specific LF design, plus cross-lingual LF reuse through the MASSIVE dataset. By adopting careful LF design pipelines and a standardized WS workflow, BoxWRENCH measures when WS remains advantageous relative to fully supervised learning via crossover points between models trained on weak labels and those trained on clean labels. The findings show that, on these more challenging tasks, WS delivers substantial gains, especially with higher-quality LFs and multilingual LF adaptation, and that simple label-models often perform robustly. The work provides extensive datasets, code, and methodological guidance to improve WS benchmarking and practical deployment across diverse real-world settings.

Abstract

Weak supervision (WS) is a popular approach for label-efficient learning, leveraging diverse sources of noisy but inexpensive weak labels to automatically annotate training data. Despite its wide usage, WS and its practical value are challenging to benchmark due to the many knobs in its setup, including: data sources, labeling functions (LFs), aggregation techniques (called label models), and end model pipelines. Existing evaluation suites tend to be limited, focusing on particular components or specialized use cases. Moreover, they often involve simplistic benchmark tasks or de-facto LF sets that are suboptimally written, producing insights that may not generalize to real-world settings. We address these limitations by introducing a new benchmark, BOXWRENCH, designed to more accurately reflect real-world usages of WS. This benchmark features tasks with (1) higher class cardinality and imbalance, (2) notable domain expertise requirements, and (3) opportunities to re-use LFs across parallel multilingual corpora. For all tasks, LFs are written using a careful procedure aimed at mimicking real-world settings. In contrast to existing WS benchmarks, we show that supervised learning requires substantial amounts (1000+) of labeled examples to match WS in many settings.

Figures (14)

• Figure 1: We compare end models trained with three different pipelines: (1) Supervised, which uses clean labels from the validation set for fine-tuning; (2) Weakly Supervised, which trains on labels obtained by applying all LFs and aggregating them with the label model, using the validation set for hyperparameter search and early stopping. Continuous-Fine-Tuning: which takes a weakly supervised model and then continuously fine-tuning it on the same clean validation labels.
• Figure 2: Using an off-the-shelf translator (DeepL) to reuse English LFs for the multilingual variants of MASSIVE18. Our approach is to apply the existing LFs written for MASSIVE18-En by first translating non-English versions of the dataset to English.
• Figure 3: Crossover points on six existing WS benchmarks: AGNews, Semeval, Yelp, ChemProt, Trec, IMDB. The crossover points for these tasks are low, less than 200 on four out of the six tasks, which is substantially lower than the crossover points in BoxWRENCH datasets.
• Figure 4: Crossover points on three of our datasets: Amazon31, Banking77, and Claude9. For both Amazon31 and Banking77, the crossover points are beyond 1,000 clean labels. For Claude9, the validation set is smaller and even training on all available examples does not result in a crossover.
• Figure 5: Crossover points on ChemProt with our new LFs. The new LFs for ChemProt demonstrate a higher crossover point when measuring F1 performance.