GreenServ: Energy-Efficient Context-Aware Dynamic Routing for Multi-Model LLM Inference

TL;DR

GreenServ tackles the high energy cost of LLM inference by routing queries to the most suitable model from a heterogeneous pool using lightweight per-query context. It formulates the problem as a contextual multi-objective optimization and solves it online with LinUCB, balancing accuracy and GPU energy consumption under latency constraints. The approach combines a Task Classifier, semantic clustering, and a complexity assessor to build a context vector that drives adaptive routing, enabling zero offline calibration and seamless model addition. Empirical results across five benchmarks and RouterBench show substantial improvements in accuracy and energy efficiency, with overhead per query remaining small, making GreenServ practical for real-world deployments. The work provides a flexible, open, and scalable framework for energy-aware LLM inference in dynamic model ecosystems.

Abstract

Large language models (LLMs) demonstrate remarkable capabilities, but their broad deployment is limited by significant computational resource demands, particularly energy consumption during inference. Static, one-model-fits-all inference strategies are often inefficient, as they do not exploit the diverse range of available models or adapt to varying query requirements. This paper presents GreenServ, a dynamic, context-aware routing framework that optimizes the trade-off between inference accuracy and energy efficiency. GreenServ extracts lightweight contextual features from each query, including task type, semantic cluster, and text complexity, and routes queries to the most suitable model from a heterogeneous pool, based on observed accuracy and energy usage. We employ a multi-armed bandit approach to learn adaptive routing policies online. This approach operates under partial feedback, eliminates the need for extensive offline calibration, and streamlines the integration of new models into the inference pipeline. We evaluated GreenServ across five benchmark tasks and a pool of 16 contemporary open-access LLMs. Experimental results show that GreenServ consistently outperforms static (single-model) and random baselines. In particular, compared to random routing, GreenServ achieved a 22% increase in accuracy while reducing cumulative energy consumption by 31%. Finally, we evaluated GreenServ with RouterBench, achieving an average accuracy of 71.7% with a peak accuracy of 75.7%. All artifacts are open-source and available as an anonymous repository for review purposes here: https://anonymous.4open.science/r/llm-inference-router-EBEA/README.md

Figures (9)

• Figure 1: The system model of GreenServ.
• Figure 2: Comparison of mean normalized accuracy (higher is better) and total energy consumption. GreenServ and contextual baselines use all available features. Error bars represent 95% confidence intervals. The static Pareto frontier in Figure \ref{['fig:a5_pareto']} is shown for reference.
• Figure 3: The left plot shows the cumulative regret over time for GreenServ and baseline MAB algorithms using the Full Features context. Shaded areas represent 95% confidence intervals. The right plot shows the moving average (window=50) regret over time for GreenServ and baseline MAB algorithms using the Full Features context, smoothing short-term fluctuations.
• Figure 4: Mean Normalized Accuracy vs. Total Energy Consumption (Wh) for different strategies at varying $\lambda$ values (0.0 to 1.0, increments of 0.2). The static Pareto front is shown for reference.
• Figure 5: Distribution of final cumulative regret across multiple runs (n=50) for different contextual feature configurations at $\lambda=0.4$.