Selective Labeling with False Discovery Rate Control

TL;DR

Conformal Labeling tackles the challenge of obtaining high-quality labels at scale by provably bounding the fraction of incorrect AI-labeled data in a selected subset through false discovery rate control. It reinterprets selective labeling as a multiple-hypothesis-testing problem and constructs conformal $p$-values by contrasting a model uncertainty score against misclassified calibration examples, then applies a data-driven BH-like threshold to form the labeled subset. The approach delivers a rigorous FDR guarantee, $\text{FDR} \le \alpha$ under mild assumptions, and demonstrates tight FDR control with high power across image, text, and LLM QA tasks, reducing labeling costs while maintaining label quality. Practically, Conformal Labeling enables reliable deployment of AI labeling by ensuring a predefined proportion of AI-assigned labels are correct on average, regardless of the underlying model performance.

Abstract

Obtaining high-quality labels for large datasets is expensive, requiring massive annotations from human experts. While AI models offer a cost-effective alternative by predicting labels, their label quality is compromised by the unavoidable labeling errors. Existing methods mitigate this issue through selective labeling, where AI labels a subset and human labels the remainder. However, these methods lack theoretical guarantees on the quality of AI-assigned labels, often resulting in unacceptably high labeling error within the AI-labeled subset. To address this, we introduce \textbf{Conformal Labeling}, a novel method to identify instances where AI predictions can be provably trusted. This is achieved by controlling the false discovery rate (FDR), the proportion of incorrect labels within the selected subset. In particular, we construct a conformal $p$-value for each test instance by comparing AI models' predicted confidence to those of calibration instances mislabeled by AI models. Then, we select test instances whose $p$-values are below a data-dependent threshold, certifying AI models' predictions as trustworthy. We provide theoretical guarantees that Conformal Labeling controls the FDR below the nominal level, ensuring that a predefined fraction of AI-assigned labels is correct on average. Extensive experiments demonstrate that our method achieves tight FDR control with high power across various tasks, including image and text labeling, and LLM QA.

Figures (4)

• Figure 1: Empirical distributions and power (employed with our method) of conformal $p$-values under different uncertainty scores. The experiments include maximum softmax probability (MSP), energy, and DOCTOR-$\alpha$ score ($D_{\alpha}$). The experiments are conducted on ImageNet with ResNet-34. The results show that both MSP and $D_{\alpha}$ score create a clear distinction between correct and incorrect predictions, thus achieving high statistical power. However, the energy score fails to provide this separation, leading to low power.
• Figure 2: Performance comparison of Conformal Labeling across models of varying accuracy. We employ Qwen3-8B, Qwen3-14B, and Qwen3-32B (model accuracy increases with parameter count) on MMLU and MMLU-Pro with $\alpha=0.1$. The results show that model with higher accuracy achieves greater power and AI-labeled ratio.
• Figure 3: Performance of Conformal Labeling with varying calibration set sizes on the MedMCQA dataset. The top row corresponds to the results from Qwen3-8B and the bottom row to those from Qwen3-32B; each column corresponds to a value of calibration ratio. Shaded regions indicate one standard deviation around the average. The results show that a large calibration set slightly reduces the variance of FDR and power, leading to a more robust selection outcome. Overall, our method is robust to changes in the calibration set size.
• Figure 4: Albation study on the selection procedure. The ablation study is conducted on the MMLU dataset with Qwen3-32B. We substitute Conformal Labeling's selection procedure with alternative selection procedures, including BH, Storey-BH, and Quantile-BH procedures. The figure shows the Conformal Labeling's selection procedure consistently achieves the tightest FDR control and the highest power among all the selection procedures.

Theorems & Definitions (4)

• Theorem 3.1
• Lemma B.1
• proof
• proof : Proof of theorem \ref{['thm:CL_fdr_iid']}