Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection

TL;DR

This work tackles routing without access to ground-truth in-domain labels by introducing Routing with Generated Data (RGD) and proposing CASCAL, a label-free router that uses consensus-based correctness estimates and hierarchical skill clustering to select expert LLMs. Empirically, routers trained on generated data—especially query-only approaches—are more robust to generator quality than query-answer baselines, with stronger generators yielding better differentiation among models. The study demonstrates that two key generator properties—being able to answer its own questions and producing questions that differentiate model performance—drive routing quality, and that carefully filtering generated data can markedly improve performance, even surpassing real-data baselines in some settings. Overall, CASCAL offers a practical pathway to robust LLM routing when ground-truth labels are unavailable and user request distributions are unknown or heterogeneous.

Abstract

Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data.

Figures (8)

• Figure 1: Overview of Routing with Generated Data (RGD). (A) Most existing routing methods require human-labeled data for skill estimation and expert selection. (B) RGD generates routing data from task descriptions, enabling both query-only and query-answer routing. (C) CASCAL extracts skills clusters from generated queries and routes to models without ground-truth labels.
• Figure 2: Overview of CASCAL. (A) Consensus Scoring: we extract model responses for each query and compute confidence-weighted consensus scores. (B) Centroid Identification: For each model and subject, we cluster queries where the model demonstrates proficiency to obtain skill centroids, then we merge similar centroids across models. (C) Cluster Ranking: we assign queries to their nearest centroid and rank models within each cluster by average consensus score. (D) Inference: we route test queries to the nearest subject and centroid, select the top-3 (or top-1) ranked models, and aggregate responses via consensus voting.
• Figure 3: Routing accuracy across RGD scenarios for Pool-Large (left) and Pool-Small (right). Colors indicate the source of routing data: validation data or data generated by different LLMs. Each bar represents the router’s average test accuracy across four datasets (MMLU-Pro, SuperGPQA, MedMCQA, and BigBench-Extra-Hard). Annotations indicate the absolute accuracy improvement of CASCAL variants over the strongest non-CASCAL baseline within the same routing family (subplot) under the same data source (color).
• Figure 4: User prompt used to generated model responses to all queries.
• Figure 5: User prompt used generate task description using Gemini-2.5-Flash.