Bayesian Orchestration of Multi-LLM Agents for Cost-Aware Sequential Decision-Making

TL;DR

This work tackles high-stakes, asymmetric decision-making with LLM agents by reframing them as probabilistic sensors within a Bayesian decision framework. It demonstrates that standard discriminative prompting cannot support sequential updating, prior correction, cost-aware actions, or VOI-driven information gathering, and proposes a generative approach that elicits per-state likelihoods $p(x|s)$ via contrastive prompts across multiple LLMs. By aggregating likelihoods with robust statistics, performing sequential Bayes updates with explicit priors, and guiding information gathering through VOI and model disagreement, the framework achieves substantial cost savings and fairness improvements in resume screening, while improving calibration. The results show a 34% cost reduction and a 45% reduction in demographic parity gaps, driven largely by multi-LLM aggregation and sequential updating, with prior correction being a critical factor. The work advocates a broader shift from discriminative to generative modeling for LLM deployment in high-stakes domains and outlines practical guidelines and avenues for future research in likelihood elicitation, VOI estimation, and cross-domain validation.

Abstract

Large language models (LLMs) are increasingly deployed as autonomous decision agents in settings with asymmetric error costs: hiring (missed talent vs wasted interviews), medical triage (missed emergencies vs unnecessary escalation), and fraud detection (approved fraud vs declined legitimate payments). The dominant design queries a single LLM for a posterior over states, thresholds "confidence," and acts; we prove this is inadequate for sequential decisions with costs. We propose a Bayesian, cost-aware multi-LLM orchestration framework that treats LLMs as approximate likelihood models rather than classifiers. For each candidate state, we elicit likelihoods via contrastive prompting, aggregate across diverse models with robust statistics, and update beliefs with Bayes rule under explicit priors as new evidence arrives. This enables coherent belief updating, expected-cost action selection, principled information gathering via value of information, and fairness gains via ensemble bias mitigation. In resume screening with costs of 40000 USD per missed hire, 2500 USD per interview, and 150 USD per phone screen, experiments on 1000 resumes using five LLMs (GPT-4o, Claude 4.5 Sonnet, Gemini Pro, Grok, DeepSeek) reduce total cost by 294000 USD (34 percent) versus the best single-LLM baseline and improve demographic parity by 45 percent (max group gap 22 to 5 percentage points). Ablations attribute 51 percent of savings to multi-LLM aggregation, 43 percent to sequential updating, and 20 percent to disagreement-triggered information gathering, consistent with the theoretical benefits of correct probabilistic foundations.

Figures (8)

• Figure 1: Total hiring costs across 1,000 resumes for different methods. Our framework (green) achieves 34% cost reduction compared to industry-standard single-LLM baseline and 19% reduction vs. best ensemble baseline. Error bars show 95% bootstrap confidence intervals.
• Figure 2: Distribution of likelihood estimates from 5 LLMs across all resumes and states. Violin plots show that models agree on typical cases but disagree substantially on 15% of resumes (wide distributions). Median aggregation (thick line) is robust to per-model outliers.
• Figure 3: Relationship between initial uncertainty (entropy after resume) and information gain from phone screening. High initial uncertainty (rightward) correlates with high entropy reduction (upward), justifying information gathering. Batch inference cannot distinguish these regions.
• Figure 4: Calibration of VOI approximation. Predicted VOI (using $\rho=0.7$) correlates $r=0.63$ with actual decision improvement from screening. Despite imperfect correlation, VOI-based screening outperforms always/never heuristics by correctly prioritizing high-uncertainty cases.
• Figure 5: Impact of prior specification on decision accuracy. Left: Empirical prior correctly concentrates probability on common states ($s_1$, $s_2$). Right: Uniform prior over-predicts rare states, causing 387 misclassifications vs. 176 with empirical prior—a 41% increase in total costs.

Theorems & Definitions (33)

• Theorem 1: Sequential Updating Impossibility
• proof
• Theorem 2: Prior Correction Impossibility
• proof
• Theorem 3: VOI Computation Impossibility
• proof
• Definition 1: Sequential Decision Problem
• Theorem 4: Sequential Updating Impossibility
• proof : Proof Sketch
• Theorem 5: Prior Correction Impossibility