DiSRouter: Distributed Self-Routing for LLM Selections

TL;DR

DiSRouter introduces a distributed self-routing framework for multi-LLM query routing, replacing a central router with a cascade of self-aware agents that decide to answer or route based on intrinsic competence. It couples a two-stage Self-Awareness Training pipeline (SFT followed by RL) with scenario adaptability to tune behavior under different cost-performance preferences, encoded as $\text{Utility} = \text{Performance} - \alpha \cdot \text{Cost}$. The approach demonstrates superior utility, robust generalization to out-of-domain tasks, and modular plug-and-play scalability, outperforming centralized and other routing baselines across diverse datasets and cost settings. Practically, DiSRouter enables flexible, cost-aware deployment of heterogeneous LLM ecosystems by leveraging intrinsic self-awareness rather than external routing judgments, with potential extensions to more complex network topologies.

Abstract

The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness, its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems.

Figures (8)

• Figure 1: Comparison between (a) centralized and (b) distributed routing. Centralized routing relies on an external router, which has limitations in adapting to new LLMs (Inflexibility) and accurately assessing the knowledge boundaries of each LLM (Unawareness). In contrast, distributed routing is not dependent on an external router and can be organized into various structures.
• Figure 2: The DiSRouter Framework.(1) The routing procedure of DiSRouter: Queries of varying difficulty enter a routing cascade ordered by LLM size/cost, where each LLM agent uses self-assessment to decide whether to answer the query or reject it and route it to the next agent. (2) The Self-Awareness Training pipeline. Green and red indicate correct and incorrect answers, respectively, while blue represents the rejection behavior ("I don't know"). (3) DiSRouter possesses scenario adaptability: At the agent level, LLMs adjust their behavior toward uncertain queries based on scenario requirements. At the system level, the overall routing distribution shifts with the scenario: a greater emphasis on cost leads to more queries being routed to smaller models.
• Figure 3: $f(\alpha) = (1-\alpha)^\gamma$
• Figure 4: System-level routing distribution.
• Figure 5: Agent-level answer rate.