SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices

TL;DR

SpecOffload addresses the bottlenecks of running large language models on memory-constrained devices by embedding speculative decoding into an offloading pipeline. It couples offline Adaptive Tensor Placement with an online ParaSpec Planner and an Interleaved Batch Pipeline to allow a draft model to reside on GPU memory and run concurrently with the target model, thereby overlapping I/O with computation. The approach yields up to 4.49x improvements in GPU core utilization and around 2.5x gains in throughput over state-of-the-art baselines, across multiple models and datasets. By revealing the importance of memory-aware scheduling and phase-specific pipelining, SpecOffload enables high-throughput LLM inference on edge-like hardware without prohibitive hardware upgrades. Through formal planning and hierarchical memory management, the method demonstrates practical gains for real-world offline batch workloads.

Abstract

Efficient LLM inference on resource-constrained devices presents significant challenges in compute and memory utilization. Due to limited GPU memory, existing systems offload model weights to CPU memory, incurring substantial I/O overhead between the CPU and GPU. This leads to two major inefficiencies: (1) GPU cores are underutilized, often remaining idle while waiting for data to be loaded; and (2) GPU memory has low impact on performance, as reducing its capacity has minimal effect on overall throughput.In this paper, we propose SpecOffload, a high-throughput inference engine that embeds speculative decoding into offloading. Our key idea is to unlock latent GPU resources for storing and executing a draft model used for speculative decoding, thus accelerating inference at near-zero additional cost. To support this, we carefully orchestrate the interleaved execution of target and draft models in speculative decoding within the offloading pipeline, and propose a planner to manage tensor placement and select optimal parameters. Compared to the best baseline, SpecOffload improves GPU core utilization by 4.49x and boosts inference throughput by 2.54x. Our code is available at https://github.com/MobiSense/SpecOffload-public .

Figures (13)

• Figure 1: GPU core utilization of SOTA methods during decoding phase. Settings: Mixtral 8x7B, Env #1, SummEval dataset, details in \ref{['sec:expsetup']}.
• Figure 2: Impact of GPU memory on throughput during decoding phase. Settings: SummEval dataset, details in \ref{['sec:expsetup']}.
• Figure 3: System overview of SpecOffload.
• Figure 4: Schematic of the decoding pipeline. At model-level, while the target model validates Batch 1, the draft model concurrently generates tokens for Batch 0 (in time slot $t_n$); the two models then alternate batches (in time slot $t_{n+1}$). At computation-level, the target model's parameters are computed layer by layer. GPU, IO, and CPU are orchestrated to perform distinct, parallelized tasks.
• Figure 5: End-to-end comparison between SpecOffload and baselines in different scenarios.