Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

Hiroki Matsutani, Naoki Matsuda, Naoto Sugiura

TL;DR

This paper introduces a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication in inference performance.

Abstract

Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.

Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

TL;DR

This paper introduces a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication in inference performance.

Abstract

Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.
Paper Structure (16 sections, 5 figures, 4 tables)

This paper contains 16 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Distributed Prompt Caching Mechanism
  4. Catalog
  5. Partial Matching
  6. False Positives
  7. Implementation
  8. Evaluations
  9. Evaluation Methodology
  10. Evaluation Results
  11. Benefit vs. Overhead
  12. Benefit of Partial Matching
  13. Benefit of Local Bloom Filter
  14. Impact of Bloom Filter False Positives
  15. Practical Considerations
  16. ...and 1 more sections

Figures (5)

  • Figure 1: System overview. A local LLM on Client 1 (left) first processes a prompt and uploads its internal states to Cache Server (middle), accelerating a subsequent query on Client 2 (right).
  • Figure 2: Data structures. The internal states and the master catalog are stored in the remote server. Local inference on the clients utilizes the cached states when available. Each local catalog is synchronized with the master on the server.
  • Figure 3: Example of catalog. Four distinct ranges of a prompt consisting of the instruction, few-shot examples, and target question are registered in the catalog.
  • Figure 4: Performance comparison between cache miss and hit cases (based on Table \ref{['tab:ttft1']}).
  • Figure 5: Performance comparison between partial matching cases on low-end setting (based on Table \ref{['tab:prefix']}).