Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When to Screen, When to Bypass: LLM-Judges in Resource-Scarce AI-Human Workflow

Ruihan Lin, Jiheng Zhang

Abstract

AI systems can generate outputs at scale, but most outputs require human approval before release. This creates a bottleneck: humans cannot keep pace with AI-generated volume. A natural response is to insert an LLM-judge that screens outputs before they reach humans, filtering errors and amplifying effective review capacity. But judges are imperfect. False rejections send correct outputs back for unnecessary rework; false acceptances consume judge capacity without relieving humans. When should outputs be routed through the judge, and when should they bypass it directly to human review? We model this workflow as a queueing network with three resource pools and use a fluid approximation to characterize optimal judge allocation. The analysis reveals that optimal allocation depends critically on which resource is the current bottleneck: screening amplifies human capacity when reviewers are scarce, yet generates a rework trap that crowds out new production when workers are stretched thin. For heterogeneous task classes with different error profiles, optimal priority can reverse across operating regimes, and classes with complementary error structures can be mixed to achieve throughput that neither class attains alone. We propose a policy that uses the fluid-optimal allocation fractions for routing and the fluid-optimal service levels for admission control, and establish its asymptotic optimality as system scale grows. Extensions incorporate human feedback that improves rework quality and joint capacity planning under budget constraints. Numerical experiments confirm rapid convergence to the fluid optimum and demonstrate that the policy significantly outperforms benchmarks that either always screen or never screen.

When to Screen, When to Bypass: LLM-Judges in Resource-Scarce AI-Human Workflow

Abstract

AI systems can generate outputs at scale, but most outputs require human approval before release. This creates a bottleneck: humans cannot keep pace with AI-generated volume. A natural response is to insert an LLM-judge that screens outputs before they reach humans, filtering errors and amplifying effective review capacity. But judges are imperfect. False rejections send correct outputs back for unnecessary rework; false acceptances consume judge capacity without relieving humans. When should outputs be routed through the judge, and when should they bypass it directly to human review? We model this workflow as a queueing network with three resource pools and use a fluid approximation to characterize optimal judge allocation. The analysis reveals that optimal allocation depends critically on which resource is the current bottleneck: screening amplifies human capacity when reviewers are scarce, yet generates a rework trap that crowds out new production when workers are stretched thin. For heterogeneous task classes with different error profiles, optimal priority can reverse across operating regimes, and classes with complementary error structures can be mixed to achieve throughput that neither class attains alone. We propose a policy that uses the fluid-optimal allocation fractions for routing and the fluid-optimal service levels for admission control, and establish its asymptotic optimality as system scale grows. Extensions incorporate human feedback that improves rework quality and joint capacity planning under budget constraints. Numerical experiments confirm rapid convergence to the fluid optimum and demonstrate that the policy significantly outperforms benchmarks that either always screen or never screen.
Paper Structure (97 sections, 18 theorems, 159 equations, 8 figures)

This paper contains 97 sections, 18 theorems, 159 equations, 8 figures.

Table of Contents

  1. Introduction
  2. When should the judge be used, and how intensively?
  3. How should judge capacity be allocated across heterogeneous tasks?
  4. Implementation and extensions.
  5. Related Literature and Positioning
  6. Human--AI collaboration.
  7. LLM judges for screening and evaluation.
  8. Queueing control under bottlenecks and rework.
  9. Paper Organization
  10. Problem Formulation
  11. Workflow Architecture
  12. Workflow overview.
  13. Rework and the judge trade-off.
  14. Quality and Error Model
  15. When does the judge improve quality?
  16. ...and 82 more sections

Key Result

Theorem 1

Fix any finite horizon $T>0$. Under Assumption ass:fluid-init and any admissible policy sequence $\{\boldsymbol{\pi}^n\}_{n \ge 1}$ with $\boldsymbol{\pi}^n \in \Pi^n$ for each $n$, the sequence of fluid-scaled processes is tight in $\mathbb{D}([0,T],\mathbb{R}^d)$ under the Skorokhod $J_1$ topology

Figures (8)

  • Figure 1: Workflow architecture. Solid arrows indicate forward flow; dashed arrows indicate rework.
  • Figure 2: Single-class optimal judge allocation $\phi^*$ and resource utilizations as human capacity $n_h$ varies. Parameters: $\lambda = 75$, $(\mu_w, \mu_j, \mu_h) = (20, 30, 10)$, $n_w = 5$, $n_j = 3$, $\alpha = 0.3$, $\beta^{(I)} = 0.1$, $\beta^{(II)} = 0.2$. Vertical lines mark theoretical thresholds.
  • Figure 3: Two-class judge allocation as $n_h$ varies. Class 1: $(\beta_1^{(I)}, \beta_1^{(II)}) = (0.05, 0.40)$ (lenient judge); Class 2: $(\beta_2^{(I)}, \beta_2^{(II)}) = (0.15, 0.10)$ (strict judge). Class 2 has higher $q_{\mathrm{acc}}$ ($k_q=2$, prioritized when humans are scarce), while Class 1 has lower $\eta$ ($k_\eta=1$, prioritized when workers are scarce). Priority reverses as $n_h$ grows, with a complementarity zone where both classes share judge capacity. Other parameters: $\lambda_i = 75$, $(\mu_w, \mu_j, \mu_h) = (20, 30, 10)$, $n_w = 10$, $n_j = 6$, $\alpha = 0.3$.
  • Figure 4: Optimal judge routing with human feedback ($\kappa=0.5$). Left: judge utilization decomposed into fresh and feedback components as human capacity varies, showing that the baseline phase structure persists but composition changes due to resource competition. Right: routing fractions $(\phi^*,\phi_{fb}^*)$, illustrating that when judge capacity saturates, the system prioritizes fresh tasks.
  • Figure 5: Optimal capacity allocation and throughput comparison. (a) Capacity allocation: as $n_h$ increases, budget shifts from judges to workers. (b) Throughput: capacity planning (solid blue) dominates fixed allocations across all $n_h$ by adapting to the prevailing bottleneck. Parameters: $B = 10$, $(\beta_1^{(I)}, \beta_1^{(II)}) = (0.05, 0.40)$, $(\beta_2^{(I)}, \beta_2^{(II)}) = (0.15, 0.10)$, $\gamma_w = \gamma_j = 1$.
  • ...and 3 more figures

Theorems & Definitions (32)

  • Theorem 1: Fluid limit
  • Lemma 1: Threshold ordering
  • Proposition 1: Four-phase structure
  • Proposition 2: Abundant workers
  • Proposition 3: Priority reversal
  • Theorem 2: Asymptotic optimality
  • Proposition 4: Feedback queue vanishes in steady state
  • Proposition 5: Judge priority under feedback
  • Theorem 3: Asymptotic optimality under feedback
  • Proposition 6: Worker-Judge coupling
  • ...and 22 more