Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing

Zhenyuan Yang, Wenxin Zheng, Mingyu Li

Abstract

GPU sharing is critical for maximizing hardware utilization in modern data centers. However, existing approaches present a stark trade-off: coarse-grained temporal multiplexing incurs severe tail-latency spikes for interactive services, while fine-grained spatial partitioning often necessitates invasive kernel modifications that compromise behavioral equivalence. We present DetShare, a novel GPU sharing system that prioritizes determinism and transparency. DetShare ensures semantic determinism (unmodified kernels yield identical results) and performance determinism (predictable tail latency), all while maintaining complete transparency (zero code modification). DetShare introduces GPU coroutines, a new abstraction that decouples logical execution contexts from physical GPU resources. This decoupling enables flexible, fine-grained resource allocation via lightweight context migration. Our evaluation demonstrates that DetShare improves training throughput by up to 79.2% compared to temporal sharing. In co-location scenarios, it outperforms state-of-the-art baselines, reducing P99 tail latency by 15.1% without compromising throughput. Furthermore, through workload-aware placement and our TPOT-First scheduling policy, DetShare decreases average inference latency by 69.1% and reduces Time-Per-Output-Token (TPOT) SLO violations by 21.2% relative to default policies.

Guaranteeing Semantic and Performance Determinism in Flexible GPU Sharing

Abstract

GPU sharing is critical for maximizing hardware utilization in modern data centers. However, existing approaches present a stark trade-off: coarse-grained temporal multiplexing incurs severe tail-latency spikes for interactive services, while fine-grained spatial partitioning often necessitates invasive kernel modifications that compromise behavioral equivalence. We present DetShare, a novel GPU sharing system that prioritizes determinism and transparency. DetShare ensures semantic determinism (unmodified kernels yield identical results) and performance determinism (predictable tail latency), all while maintaining complete transparency (zero code modification). DetShare introduces GPU coroutines, a new abstraction that decouples logical execution contexts from physical GPU resources. This decoupling enables flexible, fine-grained resource allocation via lightweight context migration. Our evaluation demonstrates that DetShare improves training throughput by up to 79.2% compared to temporal sharing. In co-location scenarios, it outperforms state-of-the-art baselines, reducing P99 tail latency by 15.1% without compromising throughput. Furthermore, through workload-aware placement and our TPOT-First scheduling policy, DetShare decreases average inference latency by 69.1% and reduces Time-Per-Output-Token (TPOT) SLO violations by 21.2% relative to default policies.
Paper Structure (58 sections, 1 equation, 8 figures, 3 tables)

This paper contains 58 sections, 1 equation, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background and Motivation
  4. AI Model Serving
  5. GPU Sharing
  6. The Sources of Semantic Non-Determinism
  7. Hardware-Level Floating-Point Non-Associativity.
  8. System-Level Serving Optimizations.
  9. Application-Level Impact: Training-Inference Mismatch.
  10. Challenges of GPU Sharing Mechanisms
  11. Challenge-#1: Semantic Determinism
  12. Formalization.
  13. Challenge-#2: Performance Determinism
  14. Theoretical Containment ($\mathcal{S}_{temporal} \subset \mathcal{S}_{spatial}$).
  15. The Reality Gap: Unmanaged Interference.
  16. ...and 43 more sections

Figures (8)

  • Figure 1: Statistical Divergence. The impact of dynamic batch fragmentation on training loss. Adapting batch sizes to fit resource fragments alters the statistical properties of the workload, leading to increased loss variance compared to the baseline.
  • Figure 2: Numerical Divergence. Absolute numerical deviation of FP16 and BF16 summations compared to their unfragmented, standalone execution. Dynamically splitting the computation grid (X-axis) inevitably alters the underlying reduction tree, exposing the non-associativity of floating-point arithmetic. This semantic divergence is especially severe for BF16.
  • Figure 3: The overall architecture of DetShare.
  • Figure 4: An example of Remapping Event and pCtx migration.
  • Figure 5: Normalized throughput of colocated model training tasks (higher is better). DetShare consistently outperforms baselines, particularly in scenarios with high resource contention or varying interference patterns.
  • ...and 3 more figures