Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL

TL;DR

This work tackles the challenge of deploying large language models on resource-constrained edge devices by introducing Splitwise, a Lyapunov-assisted DRL framework for dynamic, fine-grained edge-cloud partitioning of transformer models. By decomposing layers into attention heads and FFN sub-blocks and optimizing a drift-plus-penalty reward within a constrained MDP, Splitwise guarantees queue stability while jointly minimizing latency, energy, and accuracy loss. The method employs a hierarchical policy with dual critics in an PPO framework and online adaptation, plus partition-boundary checkpointing for fault tolerance. Experiments across diverse edge devices and model scales show substantial latency reductions ($1.4\times$–$2.8\times$), significant energy savings (up to $41\%$), and substantial $P95$ latency improvements ($53\%-61\%$) over cloud-only baselines, demonstrating practical impact for responsive and cost-efficient edge-enabled LLM inference.

Abstract

Deploying large language models (LLMs) on edge devices is challenging due to their limited memory and power resources. Cloud-only inference reduces device burden but introduces high latency and cost. Static edge-cloud partitions optimize a single metric and struggle when bandwidth fluctuates. We propose Splitwise, a novel Lyapunov-assisted deep reinforcement learning (DRL) framework for fine-grained, adaptive partitioning of LLMs across edge and cloud environments. Splitwise decomposes transformer layers into attention heads and feed-forward sub-blocks, exposing more partition choices than layer-wise schemes. A hierarchical DRL policy, guided by Lyapunov optimization, jointly minimizes latency, energy consumption, and accuracy degradation while guaranteeing queue stability under stochastic workloads and variable network bandwidth. Splitwise also guarantees robustness via partition checkpoints with exponential backoff recovery in case of communication failures. Experiments on Jetson Orin NX, Galaxy S23, and Raspberry Pi 5 with GPT-2 (1.5B), LLaMA-7B, and LLaMA-13B show that Splitwise reduces end-to-end latency by 1.4x-2.8x and cuts energy consumption by up to 41% compared with existing partitioners. It lowers the 95th-percentile latency by 53-61% relative to cloud-only execution, while maintaining accuracy and modest memory requirements.

Figures (7)

• Figure 1: Illustration of key experimental metrics, highlighting variations in (a) operational cost, (b) memory requirements, and (c) performance scaling compared to cloud-only execution. (Details of setup are in Section \ref{['sec:performance']})
• Figure 2: Comparison of model training dynamics, showing (a) training stability and (b) learning efficiency across different settings.
• Figure 3: System architecture of Splitwise. The framework consists of four main runtime components (see Section \ref{['ssub: System architecture']}): (1) Partition Controller, (2) Profiling Engine, (3) Communication Manager, and (4) Execution Runtime.
• Figure 4: Energy consumption comparison across different edge devices and LLM models on variable network conditions. Energy per request (a) on the NVIDIA Jetson Orin NX, (b) on the Samsung Galaxy S23 smartphone, and (c) on the Raspberry Pi 5.
• Figure 5: Mobile device battery life with 600 inference requests/hour.