Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs

Benjamin Kubwimana, Qijing Huang

TL;DR

EdgeReasoning investigates deploying reasoning LLMs on edge GPUs under strict latency and resource constraints. It combines empirical profiling on NVIDIA Jetson Orin with analytic latency and energy models that map token budgets to performance, and it evaluates token-control, test-time scaling, and quantization strategies to map the accuracy-latency-energy tradeoffs on edge hardware. The study shows decoding as the primary latency driver, demonstrates substantial edge-cloud cost and energy savings, and provides Pareto-frontier guidance for choosing model size, budgeting, and scaling strategy to meet diverse real-time requirements. The results enable systematic, hardware-aware deployment of reasoning LLMs at the edge with practical guidelines for balancing accuracy, speed, and cost in autonomous systems.

Abstract

Edge intelligence paradigm is increasingly demanded by the emerging autonomous systems, such as robotics. Beyond ensuring privacy-preserving operation and resilience in connectivity-limited environments, edge deployment offers significant energy and cost advantages over cloud-based solutions. However, deploying large language models (LLMs) for reasoning tasks on edge GPUs faces critical challenges from strict latency constraints and limited computational resources. To navigate these constraints, developers must balance multiple design factors - choosing reasoning versus non-reasoning architectures, selecting appropriate model sizes, allocating token budgets, and applying test-time scaling strategies - to meet target latency and optimize accuracy. Yet guidance on optimal combinations of these variables remains scarce. In this work, we present EdgeReasoning, a comprehensive study characterizing the deployment of reasoning LLMs on edge GPUs. We systematically quantify latency-accuracy tradeoffs across various LLM architectures and model sizes. We systematically evaluate prompt-based and model-tuning-based techniques for reducing reasoning token length while maintaining performance quality. We further profile test-time scaling methods with varying degrees of parallelism to maximize accuracy under strict latency budgets. Through these analyses, EdgeReasoning maps the Pareto frontier of achievable accuracy-latency configurations, offering systematic guidance for optimal edge deployment of reasoning LLMs.

EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs

TL;DR

EdgeReasoning investigates deploying reasoning LLMs on edge GPUs under strict latency and resource constraints. It combines empirical profiling on NVIDIA Jetson Orin with analytic latency and energy models that map token budgets to performance, and it evaluates token-control, test-time scaling, and quantization strategies to map the accuracy-latency-energy tradeoffs on edge hardware. The study shows decoding as the primary latency driver, demonstrates substantial edge-cloud cost and energy savings, and provides Pareto-frontier guidance for choosing model size, budgeting, and scaling strategy to meet diverse real-time requirements. The results enable systematic, hardware-aware deployment of reasoning LLMs at the edge with practical guidelines for balancing accuracy, speed, and cost in autonomous systems.

Abstract

Edge intelligence paradigm is increasingly demanded by the emerging autonomous systems, such as robotics. Beyond ensuring privacy-preserving operation and resilience in connectivity-limited environments, edge deployment offers significant energy and cost advantages over cloud-based solutions. However, deploying large language models (LLMs) for reasoning tasks on edge GPUs faces critical challenges from strict latency constraints and limited computational resources. To navigate these constraints, developers must balance multiple design factors - choosing reasoning versus non-reasoning architectures, selecting appropriate model sizes, allocating token budgets, and applying test-time scaling strategies - to meet target latency and optimize accuracy. Yet guidance on optimal combinations of these variables remains scarce. In this work, we present EdgeReasoning, a comprehensive study characterizing the deployment of reasoning LLMs on edge GPUs. We systematically quantify latency-accuracy tradeoffs across various LLM architectures and model sizes. We systematically evaluate prompt-based and model-tuning-based techniques for reducing reasoning token length while maintaining performance quality. We further profile test-time scaling methods with varying degrees of parallelism to maximize accuracy under strict latency budgets. Through these analyses, EdgeReasoning maps the Pareto frontier of achievable accuracy-latency configurations, offering systematic guidance for optimal edge deployment of reasoning LLMs.

Paper Structure

This paper contains 34 sections, 6 equations, 14 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reasoning LLMs
  4. Test-Time Scaling
  5. Reasoning Token Optimization
  6. Edge GPUs
  7. Motivation
  8. Comparison of Reasoning vs Non-Reasoning LLMs
  9. Comparison of Edge vs. Cloud Deployment
  10. Edge GPU Performance Characterization and Modeling
  11. Characterization of Latency
  12. Characterization of Power and Energy
  13. Evaluation of Inference Strategies
  14. Impact of Model Selection
  15. Impact of Reasoning Token Control Methods
  16. ...and 19 more sections

Figures (14)

  • Figure 1: Discrete accuracy-latency tradeoffs fail to capture continuous operational requirements of real-world systems like assistive robots.
  • Figure 2: Prefill latency vs. input sequence length.
  • Figure 3: Decode latency vs output and input sequence lengths.
  • Figure 4: Prefill power (left) and energy per token (right) as a function of input sequence length.
  • Figure 5: Decode phase power (left) and energy per token (right) as a function of output sequence length
  • ...and 9 more figures