APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs

TL;DR

APEX tackles the memory bottleneck of online LLM inference on constrained GPUs by enabling asynchronous CPU-GPU overlap during decode, guided by a profiling-informed scheduler. It introduces a model-driven decision process and the Asynchronous Overlap mechanism to avoid the drawbacks of batch-splitting, while selectively using Asymmetric Pipelining when beneficial. The system integrates a Llamafile-based CPU paged-attention backend and a two-thread runtime to realize sustained CPU-GPU parallelism, yielding large throughput gains with preserved latency across T4 and A10 GPUs and multiple models and workloads. This work significantly improves real-time LLM inference on edge and mid-range hardware and provides a blueprint for heterogeneous execution in memory-bound AI workloads.

Abstract

Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, particularly due to the growing KV cache during auto-regressive decoding. Hybrid GPU-CPU execution has emerged as a promising solution by offloading KV cache management and parts of attention computation to the CPU. However, a key bottleneck remains: existing schedulers fail to effectively overlap CPU-offloaded tasks with GPU execution during the latency-critical, bandwidth-bound decode phase. This particularly penalizes real-time, decode-heavy applications (e.g., chat, Chain-of-Thought reasoning) which are currently underserved by existing systems, especially under memory pressure typical of edge or low-cost deployments. We present APEX, a novel, profiling-informed scheduling strategy that maximizes CPU-GPU parallelism during hybrid LLM inference. Unlike systems relying on static rules or purely heuristic approaches, APEX dynamically dispatches compute across heterogeneous resources by predicting execution times of CPU and GPU subtasks to maximize overlap while avoiding scheduling overheads. We evaluate APEX on diverse workloads and GPU architectures (NVIDIA T4, A10), using LLaMa-2-7B and LLaMa-3.1-8B models. Compared to GPU-only schedulers like vLLM, APEX improves throughput by 84% - 96% on T4 and 11% - 89% on A10 GPUs, while preserving latency. Against the best existing hybrid schedulers, it delivers up to 72% (T4) and 37% (A10) higher throughput in long-output settings. APEX significantly advances hybrid LLM inference efficiency on such memory-constrained hardware and provides a blueprint for scheduling in heterogeneous AI systems, filling a critical gap for efficient real-time LLM applications.

Figures (9)

• Figure 1: Asymmetric pipelining split the requests into two sub-batches. The first sub-batch (red arrows) contains prefilling and GPU/CPU decoding requests. The second sub-batch (blue arrows) contains only CPU decoding requests. “pr” means pre-projection, while "po" means post-projection + FFN operations; "attn" means attention operations; "tr" means transfer of intermediate value to GPU/CPU.
• Figure 2: Latency of linear operations (Q, K, V, O projections, FFNs) for one layer of LLaMa-3.1-8B on an A10 GPU, varying with the number of tokens processed.
• Figure 3: Self-Attention computation latency (hidden size 2048 and sequence length 1024) on a V100 GPU and two AMD EPYC 7251 CPUs, by batch size. CPU computation time at batch sizes 64 and 128 are not shown due to being substantially higher than GPU latencies.
• Figure 4: APEX System Architecture
• Figure 5: Asynchronous Overlap splits the requests into two sub-batches. The first sub-batch (red arrows) contains prefilling and GPU decoding requests. The second sub-batch (blue arrows) contains only CPU decoding requests. The blue block indicates the synchronization point of GPU/CPU requests. "pr" means pre-projection, while “po” means post-projection + FFN operations; “attn” means attention operations; "tr" means transfer of intermediate value to GPU/CPU.