HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees

TL;DR

HyPAC tackles the problem of cost-efficient, multi-source data annotation with provable error control. It formulates hybrid labeling with a three-source routing scheme and trains two uncertainty thresholds using importance sampling and upper confidence bounds to guarantee the annotation error remains below a user-specified budget with high probability, while minimizing cost. The approach is distribution-free and proves both PAC-style error guarantees and cost optimality, supported by experiments showing substantial annotation cost reductions (e.g., up to 78.51% on MATH-500) across diverse datasets and uncertainty scores. This yields a principled, scalable framework for cost-quality trade-offs in data labeling, enabling practitioners to deploy efficient annotation pipelines with formal guarantees.

Abstract

Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error.

Figures (12)

• Figure 1: Hybrid annotation with three sources and two thresholds. The routing rule uses score $U(x)$ to select the appropriate annotation source: when $U(x) \le u_1$, use the non-thinking model $\tilde{f}_1$; when $u_1 < U(x) \le u_2$, use the thinking model $\tilde{f}_2$; when $U(x) > u_2$, use human annotation.
• Figure 2: Illustration of UCB-based threshold selection in the $(u_1, u_2)$ space. The green region shows the $(\epsilon, \alpha)$-PAC feasible region where $\widehat{L}_{u_1,u_2}(\alpha) \le \epsilon$ and $u_1 \le u_2$. The solid blue line is the UCB boundary, and the dashed blue line is the true risk boundary. Since the UCB upper bounds the true risk, the UCB boundary lies inside the true risk boundary. The red point marks the optimal solution selected by HyPAC, which maximizes thresholds (i.e., minimizing cost) within the feasible region.
• Figure 3: The logits-based uncertainty score achieves better cost-error tradeoff than the verbalized score. Error and cost savings of HyPAC with different uncertainty score functions under a confidence level of $\alpha = 0.05$. Experiments are conducted on MATH-500. The shaded areas represent standard deviations.
• Figure 4: HyPAC controls the annotations error while CoAnnotating fails. Comparison of annotation error rates between HyPAC and CoAnnotating li2023coannotating across three benchmarks. HyPAC controls the error rate under the specified tolerance level $0.05$ while CoAnnotating fails.
• Figure 5: HyPAC achieves higher cost savings while maintaining controlled semantic loss and error on MATH-500. Loss, cost savings, and error of HyPAC with semantic loss on MATH-500 at confidence level $\alpha=0.05$. We use Qwen3-4B-Instruct-2507 as the non-thinking model, Qwen3-4B-Thinking-2507 as the thinking model, and Qwen3-8B-Embedding as the embedding model.

Theorems & Definitions (12)

• Definition 3.1: ($\epsilon$, $\alpha$)-PAC annotation
• Lemma 4.1: Risk monotonicity
• Lemma 4.2: Cost monotonicity
• Theorem 4.3: PAC guarantee
• Theorem 4.4: Cost optimality
• proof : Proof of Lemma \ref{['lem:monotonicity']}
• proof
• proof : Proof of Theorem \ref{['thm:main']}
• Remark 1.1: Role of monotonicity in avoiding multiple testing correction
• proof