Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM as a System Service on Mobile Devices

Wangsong Yin, Mengwei Xu, Yuanchun Li, Xuanzhe Liu

TL;DR

This work addresses the challenge of maintaining persistent LLM contexts under tight memory budgets in on-device LLM-as-a-Service (LLMaaS). It introduces LLMS, a system service that decouples app memory from LLM contexts and plays memory with fine-grained, chunk-wise KV-cache management, featuring tolerance-aware compression, a swapping-recompute pipeline, and chunk lifecycle management. Evaluation on multiple devices and LLMs demonstrates up to 2 orders of magnitude reduction in context-switch latency versus baselines, with substantial gains over state-of-the-art chunk-based approaches while preserving accuracy. The results establish LLMS as a practical OS-level solution for privacy-preserving, on-device LLM-powered mobile applications.

Abstract

Being more powerful and intrusive into user-device interactions, LLMs are eager for on-device execution to better preserve user privacy. In this work, we propose a new paradigm of mobile AI: LLM as a system service on mobile devices (LLMaaS). Unlike traditional DNNs that execute in a stateless manner, such a system service is stateful: LLMs execution often needs to maintain persistent states (mainly KV cache) across multiple invocations. To minimize the LLM context switching overhead under tight device memory budget, this work presents LLMS, which decouples the memory management of app and LLM contexts with a key idea of fine-grained, chunk-wise, globally-optimized KV cache compression and swapping. By fully leveraging KV cache's unique characteristics, it proposes three novel techniques: (1) Tolerance-Aware Compression: it compresses chunks based on their measured accuracy tolerance to compression. (2) IO-Recompute Pipelined Loading: it introduces recompute to swapping-in for acceleration. (3) Chunk Lifecycle Management: it optimizes the memory activities of chunks with an ahead-of-time swapping-out and an LCTRU (Least Compression-Tolerable and Recently-Used) queue based eviction. In evaluations conducted on well-established traces and various edge devices, \sys reduces context switching latency by up to 2 orders of magnitude when compared to competitive baseline solutions.

LLM as a System Service on Mobile Devices

TL;DR

This work addresses the challenge of maintaining persistent LLM contexts under tight memory budgets in on-device LLM-as-a-Service (LLMaaS). It introduces LLMS, a system service that decouples app memory from LLM contexts and plays memory with fine-grained, chunk-wise KV-cache management, featuring tolerance-aware compression, a swapping-recompute pipeline, and chunk lifecycle management. Evaluation on multiple devices and LLMs demonstrates up to 2 orders of magnitude reduction in context-switch latency versus baselines, with substantial gains over state-of-the-art chunk-based approaches while preserving accuracy. The results establish LLMS as a practical OS-level solution for privacy-preserving, on-device LLM-powered mobile applications.

Abstract

Being more powerful and intrusive into user-device interactions, LLMs are eager for on-device execution to better preserve user privacy. In this work, we propose a new paradigm of mobile AI: LLM as a system service on mobile devices (LLMaaS). Unlike traditional DNNs that execute in a stateless manner, such a system service is stateful: LLMs execution often needs to maintain persistent states (mainly KV cache) across multiple invocations. To minimize the LLM context switching overhead under tight device memory budget, this work presents LLMS, which decouples the memory management of app and LLM contexts with a key idea of fine-grained, chunk-wise, globally-optimized KV cache compression and swapping. By fully leveraging KV cache's unique characteristics, it proposes three novel techniques: (1) Tolerance-Aware Compression: it compresses chunks based on their measured accuracy tolerance to compression. (2) IO-Recompute Pipelined Loading: it introduces recompute to swapping-in for acceleration. (3) Chunk Lifecycle Management: it optimizes the memory activities of chunks with an ahead-of-time swapping-out and an LCTRU (Least Compression-Tolerable and Recently-Used) queue based eviction. In evaluations conducted on well-established traces and various edge devices, \sys reduces context switching latency by up to 2 orders of magnitude when compared to competitive baseline solutions.
Paper Structure (23 sections, 5 equations, 20 figures, 3 tables)

This paper contains 23 sections, 5 equations, 20 figures, 3 tables.

Table of Contents

  1. Introduction
  2. LLM-as-a-Service: Motivations and Challenges
  3. On-Device Large Language Model
  4. On-Device LLM as a Mobile OS Service
  5. LLMaaS Context Managing
  6. LLMS design
  7. LLMS Overview.
  8. Tolerance-Aware Compression
  9. Swapping-Recompute Pipeline
  10. i. Profiling.
  11. ii. Planning.
  12. Chunk Lifecycle Management
  13. i. AoT Swapping.
  14. ii. LCTRU-Queue-based Eviction.
  15. Implementation and Methodology
  16. ...and 8 more sections

Figures (20)

  • Figure 1: The architecture of an LLM. We use Llama2 as an illustrative example. Embedding layer is omitted for simplicity.
  • Figure 2: The autoregressive inference with KV cache.
  • Figure 4: Memory breakdown.
  • Figure 5: Recompute overhead.
  • Figure 7: Left: a workflow of a chatbot app calling LLM Service via LLMS. Right: pseudo codes in Java.
  • ...and 15 more figures