Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

TL;DR

The paper tackles the challenge of energy-efficient on-device LLM inference by systematically evaluating 28 quantized LLM variants from the Ollama library on a Raspberry Pi 4, using hardware-based energy profiling. It pairs real-energy measurements with benchmarking across five diverse tasks to quantify energy, latency, and accuracy trade-offs of various quantization levels and techniques. Key findings include substantial energy and latency reductions from quantization, non-linear and task-dependent effects on accuracy, and Pareto-front insights that guide task-specific deployment choices for sustainable edge AI. The work bridges hardware-level energy profiling with LLM benchmarking to provide practical, actionable guidance for deploying energy-conscious LLMs in resource-constrained environments.

Abstract

Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.

Figures (15)

• Figure 1: Setup for measuring the energy consumption of LLM inference on an edge device. A Joulescope power meter is between a Raspberry Pi 4 and its power supply, with real-time energy data visualized on a laptop.
• Figure 2: Distribution of energy consumption per token for all models, all datasets included.
• Figure 3: Relationship between model size (in bytes) and energy consumption per token across different model families and benchmark datasets. Each subplot represents a specific model-dataset combination with fitted linear regression trend lines showing the scaling relationship between model size and energy consumption. Marker shapes indicate different quantization techniques.
• Figure 4: Correlation between response length and energy consumption per response.
• Figure 5: Accuracy comparison across datasets for all model variants, grouped by model family, highlighting task-specific sensitivities to model architecture and quantization.