Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Router: Prefill is All You Need

Tanay Varshney, Annie Surla, Michelle Xu, Gomathy Venkata Krishnan, Maximilian Jeblick, David Austin, Neal Vaidya, Davide Onofrio

Abstract

LLMs often share comparable benchmark accuracies, but their complementary performance across task subsets suggests that an Oracle router--a theoretical selector with perfect foresight--can significantly surpass standalone model accuracy by navigating model-specific strengths. While current routers rely on fragile semantic signals, we propose using internal prefill activations via Encoder-Target Decoupling--a functional separation between the model providing the predictive signal (the Encoder) and the model whose performance is being estimated (the Target). This allows optimized heterogeneous pairing between unique encoders and target models. We utilize Fisher Separability (J) and Effective Dimensionality (d_eff) as mathematical probes to isolate optimal layer-wise signals, providing the predictive foundation for our SharedTrunkNet architecture. SharedTrunkNet captures up to 45.58% of the accuracy gap between the strongest standalone model and the Oracle while achieving 74.31% cost savings relative to the highest-cost model.

LLM Router: Prefill is All You Need

Abstract

LLMs often share comparable benchmark accuracies, but their complementary performance across task subsets suggests that an Oracle router--a theoretical selector with perfect foresight--can significantly surpass standalone model accuracy by navigating model-specific strengths. While current routers rely on fragile semantic signals, we propose using internal prefill activations via Encoder-Target Decoupling--a functional separation between the model providing the predictive signal (the Encoder) and the model whose performance is being estimated (the Target). This allows optimized heterogeneous pairing between unique encoders and target models. We utilize Fisher Separability (J) and Effective Dimensionality (d_eff) as mathematical probes to isolate optimal layer-wise signals, providing the predictive foundation for our SharedTrunkNet architecture. SharedTrunkNet captures up to 45.58% of the accuracy gap between the strongest standalone model and the Oracle while achieving 74.31% cost savings relative to the highest-cost model.
Paper Structure (48 sections, 10 equations, 7 figures, 13 tables)

This paper contains 48 sections, 10 equations, 7 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Literature survey
  3. Non-predictive frameworks.
  4. Semantic and intent-based routing.
  5. Mechanistic routing.
  6. Strategy and multi-level orchestration.
  7. Architecture
  8. Per-LLM confidence.
  9. Per-LLM cost.
  10. Methodology
  11. LLM evaluation
  12. Semantic router
  13. Prefill router
  14. Verbosity and cost
  15. Routing function design
  16. ...and 33 more sections

Figures (7)

  • Figure 1: Overview of the two-stage routing architecture: signal extraction and confidence estimation.
  • Figure 2: Frontier pool: raw accuracy vs. total cost ($). SharedTrunkNet consistently achieves higher accuracy at lower cost than all semantic backbones, dominating the model-only baseline across the full cost range.
  • Figure 3: Frontier pool: normalized accuracy vs. normalized inverse cost. Axes anchored to pool price bounds for cross-pool comparability.
  • Figure 4: Small pool: raw accuracy vs. total cost ($).
  • Figure 5: Small pool: normalized accuracy vs. normalized inverse cost.
  • ...and 2 more figures