Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems

Amit Singh Bhatti, Vishal Vaddina, Dagnachew Birru

TL;DR

PROTEUS addresses SLA-aware routing for multi-LLM serving by learning a $\tau$-conditioned policy that translates runtime accuracy targets into routing decisions, enforced by a learned Lagrangian multiplier $\lambda$. A single model serves the full accuracy spectrum by outputting a continuous quality knob $\mu$ and using a non-linear cost term in the routing score, enabling runtime adaptation without retraining. Trained with PPO and dual updates, PROTEUS achieves high target satisfaction (floor compliance) and strong $\tau$-$\mu$ correlation, outperforming baselines on RouterBench and SPROUT while delivering near-oracle accuracy at substantial cost savings. The approach demonstrates practical, differentiable SLA navigation for large-scale LLM serving systems, with real-time adaptability and clear pathways for future extensions such as bandit feedback and latency-aware objectives.

Abstract

Production LLM deployments serve diverse workloads where cost and quality requirements vary by customer tier, time of day, and query criticality. Model serving systems accept latency SLOs directly. LLM routers do not. They force operators to tune parameters offline and guess what accuracy might result. The relationship between parameters and outcomes is indirect, non-monotonic, and dataset-dependent. Operators need to specify accuracy targets, not infer them from opaque settings. We present PROTEUS (Polymorphic Router for Operational Target Enforcement with Unified SLA), a router that accepts accuracy targets tau as runtime input. PROTEUS uses Lagrangian dual control. A learned dual variable lambda tracks constraint violations during training and conditions the policy network. This lets the router translate specified tau values into routing decisions that satisfy them. A single trained model serves the full accuracy spectrum without retraining.We evaluate on RouterBench (11 models, 405K queries) and SPROUT (14 models, 45K queries). PROTEUS achieves consistent floor compliance where accuracy meets or exceeds tau. The target-response correlation reaches 0.97 to 0.98. The closest baseline, OmniRouter, meets floors only 22% of the time despite also using Lagrangian optimization. PROTEUS operates across tau in [0.85, 0.95] from a single model. On RouterBench it achieves 90.1% accuracy, within 1.3% of oracle. On SPROUT it achieves 94.0% accuracy, within 4.6% of oracle. Cost savings reach 89.8% versus the best fixed model.

PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems

TL;DR

PROTEUS addresses SLA-aware routing for multi-LLM serving by learning a -conditioned policy that translates runtime accuracy targets into routing decisions, enforced by a learned Lagrangian multiplier . A single model serves the full accuracy spectrum by outputting a continuous quality knob and using a non-linear cost term in the routing score, enabling runtime adaptation without retraining. Trained with PPO and dual updates, PROTEUS achieves high target satisfaction (floor compliance) and strong - correlation, outperforming baselines on RouterBench and SPROUT while delivering near-oracle accuracy at substantial cost savings. The approach demonstrates practical, differentiable SLA navigation for large-scale LLM serving systems, with real-time adaptability and clear pathways for future extensions such as bandit feedback and latency-aware objectives.

Abstract

Production LLM deployments serve diverse workloads where cost and quality requirements vary by customer tier, time of day, and query criticality. Model serving systems accept latency SLOs directly. LLM routers do not. They force operators to tune parameters offline and guess what accuracy might result. The relationship between parameters and outcomes is indirect, non-monotonic, and dataset-dependent. Operators need to specify accuracy targets, not infer them from opaque settings. We present PROTEUS (Polymorphic Router for Operational Target Enforcement with Unified SLA), a router that accepts accuracy targets tau as runtime input. PROTEUS uses Lagrangian dual control. A learned dual variable lambda tracks constraint violations during training and conditions the policy network. This lets the router translate specified tau values into routing decisions that satisfy them. A single trained model serves the full accuracy spectrum without retraining.We evaluate on RouterBench (11 models, 405K queries) and SPROUT (14 models, 45K queries). PROTEUS achieves consistent floor compliance where accuracy meets or exceeds tau. The target-response correlation reaches 0.97 to 0.98. The closest baseline, OmniRouter, meets floors only 22% of the time despite also using Lagrangian optimization. PROTEUS operates across tau in [0.85, 0.95] from a single model. On RouterBench it achieves 90.1% accuracy, within 1.3% of oracle. On SPROUT it achieves 94.0% accuracy, within 4.6% of oracle. Cost savings reach 89.8% versus the best fixed model.
Paper Structure (12 sections, 3 equations, 2 figures, 4 tables)

This paper contains 12 sections, 3 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation
  4. System Architecture
  5. Constraint Enforcement via Learned Dual Variables
  6. Experiments
  7. Setup
  8. Results
  9. Runtime Adaptability: The Core Capability
  10. Standard Routing Performance
  11. Ablation Study
  12. Conclusion

Figures (2)

  • Figure 1: PROTEUS System Architecture.Top: Runtime adaptation showing how operators dynamically adjust $\tau$ based on system conditions (load spikes, SLA contracts). The same trained model serves all operating points. Middle: The inference pipeline encodes queries via DeBERTa-v3; the embedding feeds both a performance prediction head and the $\tau$-conditioned policy that outputs quality preference $\mu \in [0,1]$. The routing score combines predicted performance $p_i$, learned boosts $b_i$, and non-linear cost weighting with learnable $\gamma \in [2,8]$. Bottom: During training, batch accuracy drives Lagrangian dual updates; the feedback loop injects $\lambda$ into the policy to teach the $\tau \rightarrow \mu$ mapping.
  • Figure 2: Main Results.(a) SLA Compliance: PROTEUS (bars) consistently meets or exceeds each $\tau$ target (black lines), while baselines fail. OmniRouter (blue X markers) plateaus below targets despite per-$\tau$ training; CARROT (brown dotted lines) achieves fixed accuracy regardless of $\tau$. Green portions show accuracy above target. (b) Floor Guarantee + Cost: Achieved accuracy for RouterBench (blue) and SPROUT (orange) exceeds the floor constraint (dashed diagonal) across all $\tau$ values. Cost (dotted lines, right axis) increases with $\tau$. Shaded regions show the reliability margin. (c) Dynamic Adaptation: Four $\tau$-change scenarios showing PROTEUS tracking varying targets in real-time. Both datasets follow the target $\tau$ (dashed) with minimal lag across step, drift, cyclic, and realistic patterns.