Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models

Bhanu Prakash Vangala, Tanu Malik

TL;DR

This work tackles the self-hosted DLM dilemma by introducing Pick and Spin, a Kubernetes-based framework that jointly routes prompts to the most relevant backends and scales resources on demand. It defines a normalized multi-objective score $f(p,S_{x,y})$ with weights $w_R,w_T,w_C$ to balance relevance, latency, and cost, and combines rule-based keyword routing with a DistilBERT classifier for semantic routing. The system employs a two-dimensional deployment matrix $M \in \mathbb{R}^{L \times I}$ and an orchestration layer, Spin, that maintains warm pools and uses Little's Law for capacity planning, all managed via a unified Helm chart. Experimental evaluation across four foundation models and eight benchmarks demonstrates up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments, validating PS as a practical approach for private, scalable multi-model AI deployment. The framework thus enables cost-efficient, privacy-preserving AI at organizational scale while offering tunable trade-offs between accuracy, latency, and resource use.

Abstract

Self-hosting large language models (LLMs) is increasingly appealing for organizations seeking privacy, cost control, and customization. Yet deploying and maintaining in-house models poses challenges in GPU utilization, workload routing, and reliability. We introduce Pick and Spin, a practical framework that makes self-hosted LLM orchestration scalable and economical. Built on Kubernetes, it integrates a unified Helm-based deployment system, adaptive scale-to-zero automation, and a hybrid routing module that balances cost, latency, and accuracy using both keyword heuristics and a lightweight DistilBERT classifier. We evaluate four models, Llama-3 (90B), Gemma-3 (27B), Qwen-3 (235B), and DeepSeek-R1 (685B) across eight public benchmark datasets, with five inference strategies, and two routing variants encompassing 31,019 prompts and 163,720 inference runs. Pick and Spin achieves up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments of the same models.

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models

TL;DR

This work tackles the self-hosted DLM dilemma by introducing Pick and Spin, a Kubernetes-based framework that jointly routes prompts to the most relevant backends and scales resources on demand. It defines a normalized multi-objective score with weights to balance relevance, latency, and cost, and combines rule-based keyword routing with a DistilBERT classifier for semantic routing. The system employs a two-dimensional deployment matrix and an orchestration layer, Spin, that maintains warm pools and uses Little's Law for capacity planning, all managed via a unified Helm chart. Experimental evaluation across four foundation models and eight benchmarks demonstrates up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments, validating PS as a practical approach for private, scalable multi-model AI deployment. The framework thus enables cost-efficient, privacy-preserving AI at organizational scale while offering tunable trade-offs between accuracy, latency, and resource use.

Abstract

Self-hosting large language models (LLMs) is increasingly appealing for organizations seeking privacy, cost control, and customization. Yet deploying and maintaining in-house models poses challenges in GPU utilization, workload routing, and reliability. We introduce Pick and Spin, a practical framework that makes self-hosted LLM orchestration scalable and economical. Built on Kubernetes, it integrates a unified Helm-based deployment system, adaptive scale-to-zero automation, and a hybrid routing module that balances cost, latency, and accuracy using both keyword heuristics and a lightweight DistilBERT classifier. We evaluate four models, Llama-3 (90B), Gemma-3 (27B), Qwen-3 (235B), and DeepSeek-R1 (685B) across eight public benchmark datasets, with five inference strategies, and two routing variants encompassing 31,019 prompts and 163,720 inference runs. Pick and Spin achieves up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments of the same models.
Paper Structure (22 sections, 12 equations, 11 figures, 4 tables, 2 algorithms)

This paper contains 22 sections, 12 equations, 11 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Model Orchestration Problem
  4. Normalized Multi-Objective Score.
  5. Interpretation.
  6. Operator Profiles.
  7. Pick and Spin Framework
  8. Pick: The Routing Design
  9. Keyword Based Routing.
  10. DistilBERT Based Routing and Datasets.
  11. The Orchestrator: The Spin
  12. Matrix Representation of Deployment Options.
  13. Evaluation Metrics.
  14. Experimental Evaluation
  15. Experimental Setup and Baselines.
  16. ...and 7 more sections

Figures (11)

  • Figure 1: System architecture showing the API Gateway, Router, Orchestrator, Service Registry, and Backend Pool in the Pick and Spin framework.
  • Figure 2: Hybrid routing workflow showing the keyword based and DistilBERT based paths for complexity estimation and model selection.
  • Figure 3: Dataset distribution across eight benchmarks used for DistilBERT training and routing evaluation.
  • Figure 4: Comparison of query complexity distributions using keyword based and DistilBERT based classification. Clear separation supports relevance driven routing.
  • Figure 5: Routing success rate comparison between keyword based and DistilBERT based strategies.
  • ...and 6 more figures