Toward Sustainability-Aware LLM Inference on Edge Clusters
Kolichala Rajashekar, Nafiseh Sharghivand, Radu Prodan, Reza Farahani
TL;DR
The paper tackles the sustainability challenge of LLM inference by evaluating heterogeneous edge clusters (Jetson Orin NX and Ada 2000) and proposing carbon- and latency-aware prompt routing to balance energy use and responsiveness. It conducts benchmarking across diverse prompts and batch sizes to guide routing decisions, showing that a batch size of 4 offers the best trade-off between throughput and energy efficiency, while larger batches push memory limits. Carbon-aware routing can reduce emissions by up to about 35%, and latency-aware routing can achieve 2–3x faster end-to-end times, highlighting the value of hardware-aware workload distribution. The work provides practical guidance for edge-cloud co-design of LLM inference and suggests avenues for scalable, adaptive routing in future edge deployments.
Abstract
Large language models (LLMs) require substantial computational resources, leading to significant carbon emissions and operational costs. Although training is energy-intensive, the long-term environmental burden arises from inference, amplified by the massive global query volume. Cloud-based inference offers scalability but suffers from latency and bandwidth constraints due to centralized processing and continuous data transfer. Edge clusters instead can mitigate these limitations by enabling localized execution, yet they face trade-offs between performance, energy efficiency, and device constraints. This short paper presents a sustainability-aware LLM inference for edge clusters comprising NVIDIA Jetson Orin NX (8GB) and Nvidia Ada 2000 (16GB) devices. It aims to balance inference latency and carbon footprint through carbon- and latency-aware routing strategies, guided by empirical benchmarking of energy consumption and execution time across diverse prompts and batch (i.e., group of prompts) configurations. We compared baseline greedy strategies to carbon-aware and latency-aware strategies in prompt routing to specific hardware based on benchmarking information. Experimental evaluation shows that a batch size of four prompts achieves a trade-off between throughput, energy efficiency, while larger batches risk GPU memory saturation.