CALM: A Self-Adaptive Orchestration Approach for QoS-Aware Routing in Small Language Model based Systems

TL;DR

CALM tackles runtime uncertainties in LM-based systems by orchestrating a fleet of domain-specialized SLMs through a self-adaptive MAPE-K loop. It integrates semantic-aware routing (SORA) with a cache-enabled adaptive scheduler (CAS) to dynamically route queries, load models on demand, and manage memory, latency, and energy trade-offs. Key contributions include a formal routing scorer that fuses static semantic similarity with rolling dynamic metrics, a correlation-aware weighting scheme, a cache policy with LRU eviction, and a comprehensive experimental study showing substantial latency and energy gains over single large LLM baselines while preserving domain-specific accuracy. The work demonstrates practical deployment benefits and provides a foundation for scalable, QoS-aware LM serving in resource-constrained environments.

Abstract

AI-enabled systems are subjected to various types of runtime uncertainties, ranging from dynamic workloads, resource requirements, model drift, etc. These uncertainties have a big impact on the overall Quality of Service (QoS). This is particularly true in the case of Language Model (LM) enabled systems where the autoregressive nature of token generation introduces variability in latency, energy usage and response quality. These systems, powered by LLMs, are either resource-intensive (if run on-prem) or raise privacy/cost concerns (if leveraged using APIs). While deploying a Small Language Model (SLM) can be resource-efficient, it often falls short in addressing the diversity and scale of real-world requirements. To this, we argue that, rather than relying on any one SLM, leveraging a coordinated fleet of SLMs, each with specialized strengths can enable systems to dynamically adapt to shifting contexts and workload patterns. However, realizing the full potential of such an approach demands intelligent orchestration and continuous adaptation. To this end, we introduce CALM , a self-adaptive orchestration mechanism based on MAPE-K. Our approach continuously monitors user queries, analyzes the QoS metrics of the SLMs, identifies the optimal SLM to be used, routes the query to the identified SLM and further to enhance the effectiveness and efficiency, leverages caching and scheduling to decide the SLMs to be kept in memory. Our evaluation shows that CALM reduces latency by approximately 40% and energy consumption by 50%, while preserving domain-specific task performance when compared to single-LLM baselines.

Figures (6)

• Figure 1: CALM with Self-Adaptive Loop & Cache Module for Routing & Scheduling queries across Fleet of SLMs.
• Figure 2: Workload Distribution for a 20 minute window
• Figure 3: Variation of Latency across Candidates
• Figure 4: Tail latency analysis
• Figure 5: Average Inference Times of SLMs across examples