Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading

Minrui Xu, Dusit Niyato, Christopher G. Brinton

TL;DR

This work tackles the challenge of serving long-context LLMs in resource-constrained mobile edge networks by proposing a joint model caching and inference offloading framework powered by test-time deep reinforcement learning (T2DRL). It integrates a CoT/SC-CoT informed inference model, a Markov decision process for proactive caching/offloading, and a Double Dutch Auction mechanism to dynamically allocate edge resources while maximizing social welfare. The approach yields theoretical insights into CoT ambiguity and convergence, and empirical results showing at least a 30% reduction in system costs and improved reasoning performance across multiple datasets. Overall, the framework enables scalable, cost-efficient long-context LLM serving at the edge, with practical implications for real-world perception and reasoning tasks at mobile networks.

Abstract

Large Language Models (LLMs) can perform zero-shot learning on unseen tasks and few-shot learning on complex reasoning tasks. However, resource-limited mobile edge networks struggle to support long-context LLM serving for LLM agents during multi-round interactions with users. Unlike stateless computation offloading and static service offloading in edge computing, optimizing LLM serving at edge servers is challenging because LLMs continuously learn from context which raises accuracy, latency, and resource consumption dynamics. In this paper, we propose a joint model caching and inference offloading framework that utilizes test-time deep reinforcement learning (T2DRL) to optimize deployment and execution strategies for long-context LLM serving. In this framework, we analyze the performance convergence and design an optimization problem considering the utilization of context windows in LLMs. Furthermore, the T2DRL algorithm can learn in both the training phase and the testing phase to proactively manage cached models and service requests and adapt to context changes and usage patterns during execution. To further enhance resource allocation efficiency, we propose a double Dutch auction (DDA) mechanism, which dynamically matches supply and demand while maximizing social welfare. Finally, experimental results demonstrate that the T2DRL algorithm can reduce system costs by at least 30% compared to baselines while guaranteeing the performance of LLM agents in real-world perception and reasoning tasks.

Serving Long-Context LLMs at the Mobile Edge: Test-Time Reinforcement Learning-based Model Caching and Inference Offloading

TL;DR

This work tackles the challenge of serving long-context LLMs in resource-constrained mobile edge networks by proposing a joint model caching and inference offloading framework powered by test-time deep reinforcement learning (T2DRL). It integrates a CoT/SC-CoT informed inference model, a Markov decision process for proactive caching/offloading, and a Double Dutch Auction mechanism to dynamically allocate edge resources while maximizing social welfare. The approach yields theoretical insights into CoT ambiguity and convergence, and empirical results showing at least a 30% reduction in system costs and improved reasoning performance across multiple datasets. Overall, the framework enables scalable, cost-efficient long-context LLM serving at the edge, with practical implications for real-world perception and reasoning tasks at mobile networks.

Abstract

Large Language Models (LLMs) can perform zero-shot learning on unseen tasks and few-shot learning on complex reasoning tasks. However, resource-limited mobile edge networks struggle to support long-context LLM serving for LLM agents during multi-round interactions with users. Unlike stateless computation offloading and static service offloading in edge computing, optimizing LLM serving at edge servers is challenging because LLMs continuously learn from context which raises accuracy, latency, and resource consumption dynamics. In this paper, we propose a joint model caching and inference offloading framework that utilizes test-time deep reinforcement learning (T2DRL) to optimize deployment and execution strategies for long-context LLM serving. In this framework, we analyze the performance convergence and design an optimization problem considering the utilization of context windows in LLMs. Furthermore, the T2DRL algorithm can learn in both the training phase and the testing phase to proactively manage cached models and service requests and adapt to context changes and usage patterns during execution. To further enhance resource allocation efficiency, we propose a double Dutch auction (DDA) mechanism, which dynamically matches supply and demand while maximizing social welfare. Finally, experimental results demonstrate that the T2DRL algorithm can reduce system costs by at least 30% compared to baselines while guaranteeing the performance of LLM agents in real-world perception and reasoning tasks.
Paper Structure (23 sections, 3 theorems, 22 equations, 6 figures, 2 tables)

This paper contains 23 sections, 3 theorems, 22 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Model Serving in Mobile Edge Networks
  4. Service Caching and Offloading in Mobile Edge Networks
  5. Deep Reinforcement Learning for Edge Caching
  6. System Model
  7. Network Model
  8. CoT Inference Model
  9. Performance Convergence Analysis for CoT Prompting
  10. Cost Structure and Problem Formulation
  11. Cost Structure
  12. Test-time Reinforcement Learning-based Model Caching and Inference Offloading Algorithm
  13. Markov Decision Process
  14. Observation Space
  15. Action Space
  16. ...and 8 more sections

Key Result

Lemma 1

Given a set of examples $c_i$ with different lengths, created from the true intention $\theta^*$ and the true context $c^*$ drawn from $q_m(c)$, let $d_{i,0}$ be the initial message or task, which is derived from $q(\cdot|\theta^*_0)$ and generated from $\theta^0$ sampled from $q_m(\cdot|c^*)$. For where $\eta = 2 \frac{\epsilon(d_{i,0})}{1 - \epsilon(d_{i,0})}$ is related to the ambiguity of the

Figures (6)

  • Figure 1: Serving LLMs to handle inputs and tackle complex tasks using CoT prompting in mobile edge networks.
  • Figure 2: The T2DRL algorithm utilizes the test-time training (TTT) model in the actor-critic network.
  • Figure 3: Convergence analysis of the proposed T2DRL algorithm.
  • Figure 4: Performance comparison of the proposed T2DRL algorithm under different environment settings.
  • Figure 5: Reasoning accuracy of the T2DRL algorithm under different (a) reasoning paths; (b) vanishing factor.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Definition 1: LLMs as marginal approximations Yun2020Are
  • Definition 2: $\epsilon$-Ambiguity jiang2023latent
  • Definition 3: SC-CoT Reasoning wang2022self
  • Lemma 1
  • Theorem 1
  • proof
  • Corollary 1
  • proof