Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

SiYoung Jang, Roberto Morabito

TL;DR

The paper investigates edge-first inference for small language models, addressing cloud-latency, cost, and privacy concerns by benchmarking on single edge devices and distributed edge clusters. It introduces a metric toolkit, notably Performance-Cost Ratio (PCR) with $U = \alpha Q + (1 - \alpha) R$ and $PCR_{platform} = \frac{U}{CPR}$, to quantify trade-offs between quality, latency, and energy/cost. Key findings show domain-specialized, smaller variants can approach or match larger general models on targeted tasks and that edge deployments can achieve orders-of-magnitude lower cost per response than cloud APIs in many scenarios, though cloud fallback remains necessary for scalability. The work provides design guidance for adaptive, energy-efficient LM inference across heterogeneous environments and highlights how rate-limiting and device-aware offloading can balance edge limits with cloud re-directions in real-world workloads.

Abstract

The widespread adoption of Language Models (LMs) across industries is driving interest in deploying these services across the computing continuum, from the cloud to the network edge. This shift aims to reduce costs, lower latency, and improve reliability and privacy. Small Language Models (SLMs), enabled by advances in model compression, are central to this shift, offering a path to on-device inference on resource-constrained edge platforms. This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices, and extending to distributed edge clusters. We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity. Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems across heterogeneous environments.

Edge-First Language Model Inference: Models, Metrics, and Tradeoffs

TL;DR

The paper investigates edge-first inference for small language models, addressing cloud-latency, cost, and privacy concerns by benchmarking on single edge devices and distributed edge clusters. It introduces a metric toolkit, notably Performance-Cost Ratio (PCR) with and , to quantify trade-offs between quality, latency, and energy/cost. Key findings show domain-specialized, smaller variants can approach or match larger general models on targeted tasks and that edge deployments can achieve orders-of-magnitude lower cost per response than cloud APIs in many scenarios, though cloud fallback remains necessary for scalability. The work provides design guidance for adaptive, energy-efficient LM inference across heterogeneous environments and highlights how rate-limiting and device-aware offloading can balance edge limits with cloud re-directions in real-world workloads.

Abstract

The widespread adoption of Language Models (LMs) across industries is driving interest in deploying these services across the computing continuum, from the cloud to the network edge. This shift aims to reduce costs, lower latency, and improve reliability and privacy. Small Language Models (SLMs), enabled by advances in model compression, are central to this shift, offering a path to on-device inference on resource-constrained edge platforms. This work examines the interplay between edge and cloud deployments, starting from detailed benchmarking of SLM capabilities on single edge devices, and extending to distributed edge clusters. We identify scenarios where edge inference offers comparable performance with lower costs, and others where cloud fallback becomes essential due to limits in scalability or model capacity. Rather than proposing a one-size-fits-all solution, we present platform-level comparisons and design insights for building efficient, adaptive LM inference systems across heterogeneous environments.

Paper Structure

This paper contains 8 sections, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Benchmarking SLMs on Edge Devices
  4. General vs. Specialized Models
  5. On-device SLM inference
  6. Edge Efficiency, Cost and Quality Tradeoffs
  7. Balancing Edge Limits and Cloud Re-directions
  8. Conclusion

Figures (4)

  • Figure 1: Top: Timeline of sub-4B SLMs designed for on-device deployments. Only the smallest version per model family is shown. Bottom: Average network latency to cloud-based LLM services (ChatGPT, Claude, Perplexity) from various global regions. Error bars indicate variability.
  • Figure 2: Pass@1 accuracy of LMs with varying parameter sizes on OpenAI gsm8k. Blue bars indicate general-purpose models; the red bar (m-1.5) shows a 1.5B fine-tuned model, highlighting accuracy gains from specialization.
  • Figure 3: Top: Token generation speed (bars) and time-to-first-token (dots) on Jetson Nano (blue) and AGX (orange). The red dashed line marks human reading speed. Bottom: Power (dots) and energy per query (bars) on Nano and AGX.
  • Figure 4: Performance comparison of device selection strategies under steady and bursty workloads with the sliding window rate-limiting approach. (Left) Number of requests processed and rejected. (Middle) Tokens redirected to the cloud and associated costs. (Right) Device utilization under bursty workloads, comparing the Random and Load-Aware strategies.