Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPU-Virt-Bench: A Comprehensive Benchmarking Framework for Software-Based GPU Virtualization Systems

Jithin VG, Ditto PS

TL;DR

GPU-Virt-Bench presents a comprehensive, open-source benchmarking framework to evaluate software-based GPU virtualization across 56 metrics in 10 categories, enabling systematic comparison with MIG baselines. The framework emphasizes overhead, isolation, and LLM-specific performance to reflect multi-tenant production deployments. Empirical results on HAMi-core and BUD-FCSP reveal that software virtualization can approach a substantial portion of ideal MIG performance with actionable bottlenecks identified in memory allocation, KV caching, and fairness under contention. The work provides practical guidance for deploying multi-tenant GPU resources and sets the stage for broader multi-GPU and workflow-agnostic evaluations in future extensions.

Abstract

The proliferation of GPU-accelerated workloads, particularly in artificial intelligence and large language model (LLM) inference, has created unprecedented demand for efficient GPU resource sharing in cloud and container environments. While NVIDIA's Multi-Instance GPU (MIG) technology provides hardware-level isolation, its availability is limited to high-end datacenter GPUs. Software-based virtualization solutions such as HAMi-core and BUD-FCSP offer alternatives for broader GPU families but lack standardized evaluation methodologies. We present GPU-Virt-Bench, a comprehensive benchmarking framework that evaluates GPU virtualization systems across 56 performance metrics organized into 10 categories. Our framework measures overhead, isolation quality, LLM-specific performance, memory bandwidth, cache behavior, PCIe throughput, multi-GPU communication, scheduling efficiency, memory fragmentation, and error recovery. GPU-Virt-Bench enables systematic comparison between software virtualization approaches and ideal MIG behavior, providing actionable insights for practitioners deploying GPU resources in multi-tenant environments. We demonstrate the framework's utility through evaluation of HAMi-core, BUD-FCSP, and simulated MIG baselines, revealing performance characteristics critical for production deployment decisions.

GPU-Virt-Bench: A Comprehensive Benchmarking Framework for Software-Based GPU Virtualization Systems

TL;DR

GPU-Virt-Bench presents a comprehensive, open-source benchmarking framework to evaluate software-based GPU virtualization across 56 metrics in 10 categories, enabling systematic comparison with MIG baselines. The framework emphasizes overhead, isolation, and LLM-specific performance to reflect multi-tenant production deployments. Empirical results on HAMi-core and BUD-FCSP reveal that software virtualization can approach a substantial portion of ideal MIG performance with actionable bottlenecks identified in memory allocation, KV caching, and fairness under contention. The work provides practical guidance for deploying multi-tenant GPU resources and sets the stage for broader multi-GPU and workflow-agnostic evaluations in future extensions.

Abstract

The proliferation of GPU-accelerated workloads, particularly in artificial intelligence and large language model (LLM) inference, has created unprecedented demand for efficient GPU resource sharing in cloud and container environments. While NVIDIA's Multi-Instance GPU (MIG) technology provides hardware-level isolation, its availability is limited to high-end datacenter GPUs. Software-based virtualization solutions such as HAMi-core and BUD-FCSP offer alternatives for broader GPU families but lack standardized evaluation methodologies. We present GPU-Virt-Bench, a comprehensive benchmarking framework that evaluates GPU virtualization systems across 56 performance metrics organized into 10 categories. Our framework measures overhead, isolation quality, LLM-specific performance, memory bandwidth, cache behavior, PCIe throughput, multi-GPU communication, scheduling efficiency, memory fragmentation, and error recovery. GPU-Virt-Bench enables systematic comparison between software virtualization approaches and ideal MIG behavior, providing actionable insights for practitioners deploying GPU resources in multi-tenant environments. We demonstrate the framework's utility through evaluation of HAMi-core, BUD-FCSP, and simulated MIG baselines, revealing performance characteristics critical for production deployment decisions.
Paper Structure (106 sections, 34 equations, 1 figure, 8 tables)

This paper contains 106 sections, 34 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. The GPU Sharing Problem
  3. Virtualization Approaches
  4. The Benchmarking Gap
  5. Contributions
  6. Background and Related Work
  7. GPU Architecture Overview
  8. NVIDIA Multi-Instance GPU (MIG)
  9. Software-Based Virtualization
  10. HAMi (Heterogeneous AI Computing Virtualization Middleware)
  11. BUD-FCSP (Fine-grained Container-level SM Partitioning)
  12. Related Benchmarking Efforts
  13. Metric Taxonomy
  14. Overhead Metrics (OH-001 to OH-010)
  15. OH-001: Kernel Launch Latency
  16. ...and 91 more sections

Figures (1)

  • Figure 1: GPU-Virt-Bench Architecture (selected modules shown; additional metric modules include nccl.cu, scheduling.cu, fragmentation.cu, and error.cu)