Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

Cheng Yan, Wuyang Zhang, Zhiyuan Ning, Fan Xu, Ziyang Tao, Lu Zhang, Bing Yin, Yanyong Zhang

TL;DR

ZeroRouter introduces a universal latent space to decouple query properties from model profiling, enabling zero-shot onboarding of new LLMs and scalable routing. It combines a context-aware latent predictor, lightweight anchor-based profiling via an information-theoretic design (D-optimality), and a multi-objective ILP router to balance accuracy, cost, and latency. The approach demonstrates strong improvements on ID and OOD benchmarks, robust performance under evolving model pools, and efficient onboarding with minimal data. This framework promises cost-effective, adaptive LLM orchestration in rapidly changing model ecosystems, reducing retraining needs and improving resource utilization.

Abstract

The rapid proliferation of Large Language Models (LLMs) has led to a fragmented and inefficient ecosystem, a state of ``model lock-in'' where seamlessly integrating novel models remains a significant bottleneck. Current routing frameworks require exhaustive, costly retraining, hindering scalability and adaptability. We introduce ZeroRouter, a new paradigm for LLM routing that breaks this lock-in. Our approach is founded on a universal latent space, a model-agnostic representation of query difficulty that fundamentally decouples the characterization of a query from the profiling of a model. This allows for zero-shot onboarding of new models without full-scale retraining. ZeroRouter features a context-aware predictor that maps queries to this universal space and a dual-mode optimizer that balances accuracy, cost, and latency. Our framework consistently outperforms all baselines, delivering higher accuracy at lower cost and latency.

Breaking Model Lock-in: Cost-Efficient Zero-Shot LLM Routing via a Universal Latent Space

TL;DR

ZeroRouter introduces a universal latent space to decouple query properties from model profiling, enabling zero-shot onboarding of new LLMs and scalable routing. It combines a context-aware latent predictor, lightweight anchor-based profiling via an information-theoretic design (D-optimality), and a multi-objective ILP router to balance accuracy, cost, and latency. The approach demonstrates strong improvements on ID and OOD benchmarks, robust performance under evolving model pools, and efficient onboarding with minimal data. This framework promises cost-effective, adaptive LLM orchestration in rapidly changing model ecosystems, reducing retraining needs and improving resource utilization.

Abstract

The rapid proliferation of Large Language Models (LLMs) has led to a fragmented and inefficient ecosystem, a state of ``model lock-in'' where seamlessly integrating novel models remains a significant bottleneck. Current routing frameworks require exhaustive, costly retraining, hindering scalability and adaptability. We introduce ZeroRouter, a new paradigm for LLM routing that breaks this lock-in. Our approach is founded on a universal latent space, a model-agnostic representation of query difficulty that fundamentally decouples the characterization of a query from the profiling of a model. This allows for zero-shot onboarding of new models without full-scale retraining. ZeroRouter features a context-aware predictor that maps queries to this universal space and a dual-mode optimizer that balances accuracy, cost, and latency. Our framework consistently outperforms all baselines, delivering higher accuracy at lower cost and latency.
Paper Structure (30 sections, 19 equations, 3 figures, 2 tables)

This paper contains 30 sections, 19 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Routing
  4. Efficient Benchmarking of LLMs
  5. Methodology
  6. Universal Latent Space for Lightweight Profiling
  7. Model Inference Accuracy Estimation
  8. Model Inference Cost Estimation
  9. Model Inference Latency Estimation
  10. Context-Aware Latent Space Coordinate Prediction
  11. Hybrid Feature Representation
  12. Unified Prediction Model
  13. LLM Routing Strategies
  14. Multi-Objective Optimization.
  15. Policy-Driven Routing.
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Example of Large Language Routing Mechanism. Queries are intelligently routed to the most suitable LLM from service providers for optimal performance and cost.
  • Figure 2: An overview of the ZeroRouter framework, detailing its three core modules: (1) Latent Parameter Calibration, which establishes a universal latent space using information-theoretic anchor selection; (2) Lightweight Profiling, which efficiently maps new LLMs onto this space; and (3) Policy-Driven Routing, which predicts a query's latent coordinates and assigns it to the optimal model based on user-defined constraints for accuracy, cost, and latency.
  • Figure 3: Experimental analysis of ZeroRouter's performance and the interpretability of its latent space. (a) Real-world simulation demonstrating ZeroRouter's ability to seamlessly integrate newly released models and improve performance over time without retraining. (b) Heatmap of the difficulty vector $\boldsymbol{b}$, showing its task-agnostic nature across different semantic clusters. (c) Heatmap of the discrimination vector $\boldsymbol{\alpha}$, highlighting its task-specific variability. (d) Validation of our proposed task-aware difficulty, showing a strong monotonic correlation with the average model output token length.