Rethinking Thread Scheduling under Oversubscription: A User-Space Framework for Coordinating Multi-runtime and Multi-process Workloads

TL;DR

This work addresses oversubscription in modern HPC/AI workloads that combine multiple runtimes by introducing USF, a seamless user-space scheduling framework embedded in glibc, and SCHED_COOP, a cooperative policy that minimizes preemption. The approach leverages the nOS-V runtime to convert pthreads into tasks managed by a centralized scheduler, enabling cross-process coordination without application modifications. Empirical evaluations across nested runtimes, multi-process co-execution, and realistic workloads (microservices and MD simulations) show gains up to 2.4x and demonstrate that seamless user-space scheduling can unlock complex runtime compositions while preserving TLS and compatibility with existing runtimes. The work highlights the practical impact of allowing user-space control over scheduling decisions to reduce interference, improve throughput, and simplify the integration of heterogeneous software stacks in orders of magnitude more complex HPC/AI workloads.

Abstract

The convergence of high-performance computing (HPC) and artificial intelligence (AI) is driving the emergence of increasingly complex parallel applications and workloads. These workloads often combine multiple parallel runtimes within the same application or across co-located jobs, creating scheduling demands that place significant stress on traditional OS schedulers. When oversubscribed (there are more ready threads than cores), OS schedulers rely on periodic preemptions to multiplex cores, often introducing interference that may degrade performance. In this paper, we present: (1) The User-space Scheduling Framework (USF), a novel seamless process scheduling framework completely implemented in user-space. USF enables users to implement their own process scheduling algorithms without requiring special permissions. We evaluate USF with its default cooperative policy, (2) SCHED_COOP, designed to reduce interference by switching threads only upon blocking. This approach mitigates well-known issues such as Lock-Holder Preemption (LHP), Lock-Waiter Preemption (LWP), and scalability collapse. We implement USF and SCHED_COOP by extending the GNU C library with the nOS-V runtime, enabling seamless coordination across multiple runtimes (e.g., OpenMP) without requiring invasive application changes. Evaluations show gains up to 2.4x in oversubscribed multi-process scenarios, including nested BLAS workloads, multi-process PyTorch inference with LLaMA-3, and Molecular Dynamics (MD) simulations.

Figures (5)

• Figure 1: Glibcv architecture diagram. Application's standard API calls are forwarded to the USF backend if enabled, which bridges with the nOS-V API. nOS-V schedules threads according to the selected policy.
• Figure 2: Evaluated matmul software stacks. a) Baseline with yield. b) Manual nOS-V integration. c) Seamless nOS-V integration. d) Unmodified (no yield).
• Figure 3: Evaluation of matmul with two nested runtimes. Figure a) numbers show performance (higher is better), while b), c), and d) show element-wise speedup with respect to a). The colors of all heatmaps show performance, not speedup. White squares are timed out experiments (> 15 minutes). Squares without speedup lack the Baseline reference because of time out.
• Figure 4: Evaluation of the agentic microservices benchmark. The top figure shows latency and throughput as the frequency of requests increases. The bottom figure shows the detailed per-request start and end time of the test case with rate 0.33.
• Figure 5: LAMMPS and DeePMD-Kit evaluation with two ensembles for several configurations. All ensembles use 56 MPI and 2 OpenMP each, except co-location versions that limit MPI ranks to 28 in order to avoid oversubscription. Figure a) shows performance. Figure b) shows measured total main memory bandwidth for both reads and writes. The inner plot in Figure b) shows a zoomed area of 20s starting at second 150. Average bandwidth in GB/s per scenario is: Exclusive 165.36, colocation_node 193.09, colocation_socket 185.80, coexecution_node 208.17, coexecution_socket 201.99, schedcoop_node 214.78, schedcoop_socket 205.64.