Efficiently Executing High-throughput Lightweight LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management

TL;DR

The paper tackles the inefficiency of running lightweight LLM inferences on heterogeneous opportunistic GPU clusters by decoupling LLM context from inferences and persisting contexts on GPUs (Pervasive Context Management). It demonstrates how transforming a context-agnostic fact-verification workflow into a context-aware version using a two-step load-context and infer-process achieves substantial end-to-end speedups and robust performance under preemption and resource variability. Through a PfF case study built on Parsl-TaskVine, the authors show up to a 72.1% reduction in runtime on static resources and scalable opportunistic execution on tens of percent of cluster GPUs, with further gains under aggressive preemption and dynamic capacity. The approach amortizes the expensive model loading cost, reduces batch-size tuning sensitivity, and enables rapid rebound when GPUs rejoin the pool, offering practical impact for LLM-enabled HPC workflows. Limitations include applicability to lightweight models that fit on a single node and additional management overhead, suggesting directions for broader model scales and optimized context transfer.

Abstract

The rise of Generative AI introduces a new class of HPC workloads that integrates lightweight LLMs with traditional high-throughput applications to accelerate scientific discovery. The current design of HPC clusters is inadequate to support this new class however, either incurring long wait times on static batch queues or repeatedly paying expensive LLM startup costs upon resource preemption. To circumvent both the long queues and high startup costs, we propose to "decouple" the LLM initialization context from the actual LLM inferences, and retain the context in GPUs until it is no longer needed, a technique we term "Pervasive Context Management". We transform a fact verification application to enable this technique, allowing it to reduce its execution time by 72.1% (from 3 hours to 48 minutes) using the same amount of GPUs, and scale opportunistically on 32.8% of all GPUs in the cluster and further reduce the execution time to 13 minutes.

Figures (9)

• Figure 1: Different Resource Allocation Models
• Figure 2: Overview of the Parsl-TaskVine Framework
• Figure 3: Code Example of a Context-agnostic Fact Verification Application
• Figure 4: Overview of Pervasive Context Management
• Figure 5: Code Example of a Context-aware LLM Inference Application