Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge
Maximilian Abstreiter, Sasu Tarkoma, Roberto Morabito
TL;DR
This study systematically evaluates the feasibility of on-device generative LM inference on two edge platforms (Raspberry Pi 5 CPU and Jetson Orin Nano GPU) across 11 models and two quantization schemes. By measuring memory footprint, latency, energy, throughput, usability, and cost, the authors quantify the trade-offs between model size, quantization, and hardware capabilities, revealing memory and energy bottlenecks that persist even with compression. They introduce a metric S_t = E/t to capture the throughput-energy trade-off and show that GPU inference provides higher throughput while incurring longer load times and greater memory usage, whereas CPU inference is more memory-constrained and sensitive to threading and power modes. The findings suggest edge inference is promising for small models and latency-sensitive applications, with meaningful cost advantages over cloud APIs in many scenarios, but large models remain challenging; ongoing hardware optimizations (e.g., GenAI-optimized NPUs) and memory-efficient model designs will be key to broader edge deployment. Key contributions include a cross-device, multi-model benchmark of edge LM inference, an analysis of quantization effects on memory, latency, and energy, a detailed examination of throughput-energy trade-offs, a qualitative assessment of model performance, and a monetized cost comparison that highlights potential edge-cost savings over API-based cloud services.
Abstract
The rapid rise of Language Models (LMs) has expanded the capabilities of natural language processing, powering applications from text generation to complex decision-making. While state-of-the-art LMs often boast hundreds of billions of parameters and are primarily deployed in data centers, recent trends show a growing focus on compact models-typically under 10 billion parameters-enabled by techniques such as quantization and other model compression techniques. This shift paves the way for LMs on edge devices, offering potential benefits such as enhanced privacy, reduced latency, and improved data sovereignty. However, the inherent complexity of even these smaller models, combined with the limited computing resources of edge hardware, raises critical questions about the practical trade-offs in executing LM inference outside the cloud. To address these challenges, we present a comprehensive evaluation of generative LM inference on representative CPU-based and GPU-accelerated edge devices. Our study measures key performance indicators-including memory usage, inference speed, and energy consumption-across various device configurations. Additionally, we examine throughput-energy trade-offs, cost considerations, and usability, alongside an assessment of qualitative model performance. While quantization helps mitigate memory overhead, it does not fully eliminate resource bottlenecks, especially for larger models. Our findings quantify the memory and energy constraints that must be considered for practical real-world deployments, offering concrete insights into the trade-offs between model size, inference performance, and efficiency. The exploration of LMs at the edge is still in its early stages. We hope this study provides a foundation for future research, guiding the refinement of models, the enhancement of inference efficiency, and the advancement of edge-centric AI systems.