Balancing Sustainability And Performance: The Role Of Small-Scale Llms In Agentic Artificial Intelligence Systems

TL;DR

This work tackles the energy burden of LLM inference in agentic AI by conducting a tri-objective assessment over $EnvironmentalImpact$, $UserExperience$, and $OutputQuality$ across a wide set of open-weights LLMs versus GPT-4o in real-world deployment. Using the ML-Energy Benchmark and a multi-agent validation framework, it shows that carefully chosen smaller open-weight models (e.g., Qwen 3 variants) can achieve near-parity in output quality with substantial energy savings (up to around 70% per request) and shorter or competitive decode latency. Quantization (4-bit) and distillation offer additional energy reductions, though effects on latency and quality are model- and task-dependent; mixture-of-experts architectures can further reduce active parameters during inference. The study provides actionable guidelines for sustainable AI design, including batch-size and GPU allocation strategies, and presents a replicable benchmarking approach to balance efficiency with performance in enterprise-scale AI agents.

Abstract

As large language models become integral to agentic artificial intelligence systems, their energy demands during inference may pose significant sustainability challenges. This study investigates whether deploying smaller-scale language models can reduce energy consumption without compromising responsiveness and output quality in a multi-agent, real-world environments. We conduct a comparative analysis across language models of varying scales to quantify trade-offs between efficiency and performance. Results show that smaller open-weights models can lower energy usage while preserving task quality. Building on these findings, we propose practical guidelines for sustainable artificial intelligence design, including optimal batch size configuration and computation resource allocation. These insights offer actionable strategies for developing scalable, environmentally responsible artificial intelligence systems.

Figures (8)

• Figure 1: Workflow of a multi-agent framework interacting with a client.
• Figure 2: Output Quality versus Energy Consumption of the Best-performing Open-Weights LLMs. Scores shown for the lowest-energy configuration among tested batch sizes. Colors denote LLM families, while symbols represent model sizes. To simplify visualization, a single symbol may correspond to multiple closely related sizes (e.g., circles for 10B, 12B, and 14B; crosses for 7B and 8B).
• Figure 3: Ranking of models under different weighting scenarios. The y-axis represents the rank of each model, while the table indicates the corresponding weighting scenario ($w_{Quality}$, $w_{Energy}$ and $w_{Latency}$). Node colors represent different LLMs, and the numbers inside each node indicate their overall metrics. The baseline GPT-4o is shown with solid lines, while evaluated models are shown with dotted lines.
• Figure 4: Energy consumption (Joules, log scale) and decode latency (seconds) per request across Qwen 2.5 model sizes (0.5B - 72B), with corresponding F1-scores shown above each group. The x-axis represents model size in billions of parameters, while the left y-axis shows energy per request (blue) and the right y-axis shows decode latency per request (orange). The variation in box sizes across models relates to differences in batch size.
• Figure 5: Estimated VRAM Requirements Across Model Sizes and GPU Configurations for Qwen 2.5. The percentage above each bar shows the ratio of overhead relative to total VRAM usage.