ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision

TL;DR

This work addresses the noise inherent in weak supervision by introducing ULF, a cross-validation–driven method that re-estimates the LF-to-class mapping using out-of-sample predictions on LF-guided folds. The core idea is to leverage the LF-match structure captured in matrices $Z$ and $T$, producing a refined mapping $T^*$ through $T^* = p \hat{Q} + (1 - p) T$, and to generate cleaner labels $Y^* = Z T^*$ without requiring manual labels. The approach includes handling unlabeled samples via a labeling rate $\lambda$ and an optional Cosine self-training step, and it calibrates LF-class confidences via $\hat{Q}_{L \times K}$ to better reflect data proportions. Empirical results across six English WS datasets and two African languages show ULF frequently surpassing strong baselines, demonstrating the practical value of LF-aware cross-validation in denoising weak labels for robust downstream learning.

Abstract

A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs), rule-based mechanisms that generate artificial labels for the associated classes. In this work, we investigate noise reduction techniques for WS based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for Unsupervised Labeling Function correction, which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment on highly reliable cross-validated samples. Evaluation on multiple datasets confirms ULF's effectiveness in enhancing WS learning without the need for manual labeling.

Figures (3)

• Figure 1: ULF. Noisy training labels $y*$ are obtained by multiplying the matrices $Z$ and $T$. The most confident predictions $\hat{y}$ are calculated using $k$-fold cross-validation and used to estimate new LFs-to-class correspondence and update the T matrix. The clean labels are obtained by multiplying the updated $T$ and $Z$ matrices.
• Figure 2: Weak annotation encoded with Z and T matrices, with Z containing LFs matches in samples, and T representing LF-to-class mapping. $T*$ is an improved version of $T$ with ULF. Applying $Z$ and $T$ (or $T*$) matrices multiplication and majority vote yields labels $Y$.
• Figure 3: Transformation of T matrix with ULF after first and second iterations.