Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Elastic On-Device LLM Service

Wangsong Yin, Rongjie Yi, Daliang Xu, Gang Huang, Mengwei Xu, Xuanzhe Liu

TL;DR

This paper tackles the challenge of running large language models on mobile devices while meeting diverse latency requirements. It introduces ElastiLM, which elastifies both the model and the prompt to adapt to per-request SLOs, through one-shot reordering of permutation-consistent Transformer units and a dual-head Tiny Language Model that orchestrates prompt-model elasticity. Offline preparation yields high-quality sub-models with minimal online switching, and a lightweight TLM guides prompt compression and model selection to satisfy latency constraints without sacrificing accuracy. Empirical results on multiple devices and LLMs show substantial accuracy gains (up to 14.83 percentage points, average 10.45%), with low switching overhead (<1% relative TTFT impact) and feasible offline GPU-hour requirements, demonstrating practical deployment potential for elastic on-device LLM services.

Abstract

On-device Large Language Models (LLMs) are transforming mobile AI, catalyzing applications like UI automation without privacy concerns. Nowadays the common practice is to deploy a single yet powerful LLM as a general task solver for multiple requests. We identify a key system challenge in this paradigm: current LLMs lack the elasticity to serve requests that have diversified Service-Level Objectives (SLOs) on inference latency. To tackle this, we present \sys, an on-device LLM service that elasticizes both the model and the prompt dimension of a full LLM. It incorporates (1) a one-shot neuron-reordering method, which leverages the intrinsic permutation consistency in transformer models to generate high-quality elasticized sub-models with minimal runtime switching overhead; (2) a dual-head tiny language model, which efficiently and effectively refines the prompt and orchestrates the elastification between model and prompt. We implement such an elastic on-device LLM service on multiple COTS smartphones, and evaluate \sys on both standalone NLP/mobile-agent datasets and end-to-end synthesized traces. On diverse SLOs, \sys outperforms 7 strong baselines in (absolute) accuracy by up to 14.83\% and 10.45\% on average, with <1\% TTFT switching overhead, on-par memory consumption and <100 offline GPU hours.

Elastic On-Device LLM Service

TL;DR

This paper tackles the challenge of running large language models on mobile devices while meeting diverse latency requirements. It introduces ElastiLM, which elastifies both the model and the prompt to adapt to per-request SLOs, through one-shot reordering of permutation-consistent Transformer units and a dual-head Tiny Language Model that orchestrates prompt-model elasticity. Offline preparation yields high-quality sub-models with minimal online switching, and a lightweight TLM guides prompt compression and model selection to satisfy latency constraints without sacrificing accuracy. Empirical results on multiple devices and LLMs show substantial accuracy gains (up to 14.83 percentage points, average 10.45%), with low switching overhead (<1% relative TTFT impact) and feasible offline GPU-hour requirements, demonstrating practical deployment potential for elastic on-device LLM services.

Abstract

On-device Large Language Models (LLMs) are transforming mobile AI, catalyzing applications like UI automation without privacy concerns. Nowadays the common practice is to deploy a single yet powerful LLM as a general task solver for multiple requests. We identify a key system challenge in this paradigm: current LLMs lack the elasticity to serve requests that have diversified Service-Level Objectives (SLOs) on inference latency. To tackle this, we present \sys, an on-device LLM service that elasticizes both the model and the prompt dimension of a full LLM. It incorporates (1) a one-shot neuron-reordering method, which leverages the intrinsic permutation consistency in transformer models to generate high-quality elasticized sub-models with minimal runtime switching overhead; (2) a dual-head tiny language model, which efficiently and effectively refines the prompt and orchestrates the elastification between model and prompt. We implement such an elastic on-device LLM service on multiple COTS smartphones, and evaluate \sys on both standalone NLP/mobile-agent datasets and end-to-end synthesized traces. On diverse SLOs, \sys outperforms 7 strong baselines in (absolute) accuracy by up to 14.83\% and 10.45\% on average, with <1\% TTFT switching overhead, on-par memory consumption and <100 offline GPU hours.
Paper Structure (19 sections, 2 equations, 33 figures, 3 tables)

This paper contains 19 sections, 2 equations, 33 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Motivations
  3. Elastic on-device LLM service
  4. Opportunities and challenges
  5. ElastiLM Design
  6. Overview
  7. Model elastification
  8. Prompt elastification
  9. Implementation
  10. Evaluation
  11. Experimental settings
  12. End-to-end performance
  13. Performance on standalone datasets
  14. Offline stage overhead
  15. Sensitivity analysis
  16. ...and 4 more sections

Figures (33)

  • Figure 1: The system model. The service can be upgraded/downgraded at runtime to adapt to various SLOs.
  • Figure 2: Prefill latency.
  • Figure 3: Decode latency.
  • Figure 5: Elastic LLM does not translate to Elastic LLM service. Non-negligible request-level overhead of data movement is still suffered when directly employing model pruning.
  • Figure 6: Prompt elastification.
  • ...and 28 more figures