Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment

Warren Johnson, Charles Lee

Abstract

Selecting the appropriate model at inference time -- the routing problem -- requires jointly optimizing output quality, cost, latency, and governance constraints. Existing approaches delegate this decision to LLM-based classifiers or preference-trained routers that are themselves costly and high-latency, reducing a multi-objective optimization to single-dimensional quality prediction. We argue that small language models (SLMs, 1-4B parameters) have now achieved sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification, potentially making the routing decision negligible in the inference budget. We test this thesis on a six-label taxonomy through two studies. Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus. Qwen-2.5-3B achieves the best exact-match accuracy (0.783), the strongest latency-accuracy tradeoff, and the only nonzero accuracy on all six task families. Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm, comparing Phi-4-mini, Qwen-2.5-3B, and DeepSeek-V3 against a no-routing control. DeepSeek-V3 attains the highest accuracy (0.830) but fails the pre-registered P95 latency gate (2,295 ms); Qwen-2.5-3B is Pareto-dominant among self-hosted models (0.793 accuracy, 988 ms median, $0 marginal cost). No model meets the standalone viability criterion (>=0.85 accuracy, <=2,000 ms P95). The cost and latency prerequisites for SLM-based routing are met; the accuracy gap of 6-8 percentage points and the untested question of whether correct classification translates to downstream output quality bound the remaining distance to production viability.

Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment

Abstract

Selecting the appropriate model at inference time -- the routing problem -- requires jointly optimizing output quality, cost, latency, and governance constraints. Existing approaches delegate this decision to LLM-based classifiers or preference-trained routers that are themselves costly and high-latency, reducing a multi-objective optimization to single-dimensional quality prediction. We argue that small language models (SLMs, 1-4B parameters) have now achieved sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification, potentially making the routing decision negligible in the inference budget. We test this thesis on a six-label taxonomy through two studies. Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus. Qwen-2.5-3B achieves the best exact-match accuracy (0.783), the strongest latency-accuracy tradeoff, and the only nonzero accuracy on all six task families. Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm, comparing Phi-4-mini, Qwen-2.5-3B, and DeepSeek-V3 against a no-routing control. DeepSeek-V3 attains the highest accuracy (0.830) but fails the pre-registered P95 latency gate (2,295 ms); Qwen-2.5-3B is Pareto-dominant among self-hosted models (0.793 accuracy, 988 ms median, $0 marginal cost). No model meets the standalone viability criterion (>=0.85 accuracy, <=2,000 ms P95). The cost and latency prerequisites for SLM-based routing are met; the accuracy gap of 6-8 percentage points and the untested question of whether correct classification translates to downstream output quality bound the remaining distance to production viability.

Paper Structure

This paper contains 17 sections, 1 theorem, 5 equations, 1 figure, 9 tables.

Table of Contents

  1. Introduction
  2. The Model Selection Problem for Front-Door Routing
  3. Beyond Classification: The Multi-Objective Routing Gap
  4. Research Questions and Hypotheses
  5. Methods: Study 1 (Harmonized Offline Benchmark)
  6. Results: Study 1 (Harmonized Benchmark)
  7. Study 2: Four-Arm Randomized Experiment Design
  8. Results: Study 2 (Four-Arm Randomized Experiment)
  9. Discussion
  10. Closing the Gap: Planned Remediation Paths
  11. Threats to Validity
  12. Reproducibility
  13. Conclusion
  14. Complete Run IDs and Software Versions
  15. Study 1: Harmonized Offline Benchmark (Azure T4)
  16. ...and 2 more sections

Key Result

Proposition 1

Arm $i$ dominates arm $j$ if and only if with at least one strict inequality. Arms on the Pareto frontier are those not dominated by any other arm.$\blacktriangleleft$$\blacktriangleleft$

Figures (1)

  • Figure 1: Accuracy--P95 latency scatterplot for all evaluated models. Both axes now use the pre-registered gating metrics: exact-match accuracy ($y$) and P95 latency ($x$). Filled markers: Study 1 harmonized benchmark (identical hardware). Open markers ($^{\text{E}}$): Study 2 randomized experiment ($n_{\text{eff}}\!=\!60$). The shaded green region marks the viable operating zone (accuracy $\geq 0.85$, P95 $\leq 2{,}000$ ms). No model enters the viable region. DeepSeek-V3 approaches the accuracy threshold but exceeds the latency gate ($P95 = 2{,}295$ ms). Qwen-2.5-3B is Pareto-dominant among self-hosted models.

Theorems & Definitions (1)

  • Proposition 1: Pareto Dominance