Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multi-Instance GPU Efficiency via Sub-Entry Sharing TLB Design

Bingyao Li, Yueqi Wang, Tianyu Wang, Lieven Eeckhout, Jun Yang, Aamer Jaleel, Xulong Tang

TL;DR

The paper tackles performance degradation in MIG-enabled GPUs caused by a shared L3 TLB across instances. It introduces STAR, a hardware-supported mechanism that allows two base addresses to share L3 TLB entries by partitioning sub-entries and dynamically switching between shared and non-shared states based on demand. Through detailed analysis and evaluation, STAR demonstrates substantial gains averaging around 30% across multi-tenant workloads and up to ~51% in peak cases, with modest hardware overhead. The approach is shown to improve L3 TLB hit rates and sub-entry utilization, and it remains complementary to other TLB optimizations such as MASK and static partitioning. Collectively, STAR enhances MIG scalability for diverse co-running applications by mitigating TLB thrashing and efficiently utilizing sub-entry resources.

Abstract

NVIDIA's Multi-Instance GPU (MIG) technology enables partitioning GPU computing power and memory into separate hardware instances, providing complete isolation including compute resources, caches, and memory. However, prior work identifies that MIG does not extend to partitioning the last-level TLB (i.e., L3 TLB), which remains shared among all instances. To enhance TLB reach, NVIDIA GPUs reorganized the TLB structure with 16 sub-entries in each L3 TLB entry that have a one-to-one mapping to the address translations for 16 pages of size 64KB located within the same 1MB aligned range. Our comprehensive investigation of address translation efficiency in MIG identifies two main issues caused by L3 TLB sharing interference: (i) it results in performance degradation for co-running applications, and (ii) TLB sub-entries are not fully utilized before eviction. Based on this observation, we propose STAR to improve the utilization of TLB sub-entries through dynamic sharing of TLB entries across multiple base addresses. STAR evaluates TLB entries based on their sub-entry utilization to optimize address translation storage, dynamically adjusting between a shared and non-shared status to cater to current demand. We show that STAR improves overall performance by an average of 30.2% across various multi-tenant workloads.

Improving Multi-Instance GPU Efficiency via Sub-Entry Sharing TLB Design

TL;DR

The paper tackles performance degradation in MIG-enabled GPUs caused by a shared L3 TLB across instances. It introduces STAR, a hardware-supported mechanism that allows two base addresses to share L3 TLB entries by partitioning sub-entries and dynamically switching between shared and non-shared states based on demand. Through detailed analysis and evaluation, STAR demonstrates substantial gains averaging around 30% across multi-tenant workloads and up to ~51% in peak cases, with modest hardware overhead. The approach is shown to improve L3 TLB hit rates and sub-entry utilization, and it remains complementary to other TLB optimizations such as MASK and static partitioning. Collectively, STAR enhances MIG scalability for diverse co-running applications by mitigating TLB thrashing and efficiently utilizing sub-entry resources.

Abstract

NVIDIA's Multi-Instance GPU (MIG) technology enables partitioning GPU computing power and memory into separate hardware instances, providing complete isolation including compute resources, caches, and memory. However, prior work identifies that MIG does not extend to partitioning the last-level TLB (i.e., L3 TLB), which remains shared among all instances. To enhance TLB reach, NVIDIA GPUs reorganized the TLB structure with 16 sub-entries in each L3 TLB entry that have a one-to-one mapping to the address translations for 16 pages of size 64KB located within the same 1MB aligned range. Our comprehensive investigation of address translation efficiency in MIG identifies two main issues caused by L3 TLB sharing interference: (i) it results in performance degradation for co-running applications, and (ii) TLB sub-entries are not fully utilized before eviction. Based on this observation, we propose STAR to improve the utilization of TLB sub-entries through dynamic sharing of TLB entries across multiple base addresses. STAR evaluates TLB entries based on their sub-entry utilization to optimize address translation storage, dynamically adjusting between a shared and non-shared status to cater to current demand. We show that STAR improves overall performance by an average of 30.2% across various multi-tenant workloads.
Paper Structure (22 sections, 17 figures, 4 tables, 2 algorithms)

This paper contains 22 sections, 17 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Multi-Instance GPU
  4. Address Translation in MIG
  5. Methodology
  6. Baseline Configuration
  7. Applications
  8. Quantitative Analysis of MIG Multi-Tenancy
  9. Overall Performance Characteristics
  10. Sub-Entry Utilization Characterization
  11. Sub-Entry Sharing-Aware TLB
  12. Sub-Entry Sharing-Aware TLB Format
  13. TLB Lookup and Insertion Process
  14. Hardware Overhead
  15. Evaluation
  16. ...and 7 more sections

Figures (17)

  • Figure 1: Performance of co-running applications on NVIDIA's A100.
  • Figure 2: TLB structure and address translation process in A100.
  • Figure 3: The normalized performance of each application within the workload.
  • Figure 4: CDF of translation reuse distances at the L3 TLB.
  • Figure 5: CDF of TLB sub-entry utilization when running in isolation.
  • ...and 12 more figures