Peformance Isolation for Inference Processes in Edge GPU Systems

TL;DR

This work systematically compares NVIDIA GPU isolation mechanisms—MPS, MIG, and Green Contexts (GC)—to achieve predictable inference timing for safety-critical deep learning tasks on both general-purpose A100 GPUs and edge devices (Jetson Orin Nano/AGX). It introduces maximum inference frequency search and IMS-based performance benchmarks to quantify temporal guarantees under contention, revealing MIG as the most robust isolation method, MPS as a practical but non-isolating option, and GC as a promising low-overhead alternative with limitations in memory isolation. Across platforms, compute-bound models tend to retain higher IMS under resource reductions, while memory-bound models are more sensitive to contention, underscoring platform- and model-dependent trade-offs. The results suggest a path toward more flexible, memory-aware, and dynamically managed GPU isolation to support reliable, real-time DL inference in embedded and safety-critical systems.

Abstract

This work analyzes the main isolation mechanisms available in modern NVIDIA GPUs: MPS, MIG, and the recent Green Contexts, to ensure predictable inference time in safety-critical applications using deep learning models. The experimental methodology includes performance tests, evaluation of partitioning impact, and analysis of temporal isolation between processes, considering both the NVIDIA A100 and Jetson Orin platforms. It is observed that MIG provides a high level of isolation. At the same time, Green Contexts represent a promising alternative for edge devices by enabling fine-grained SM allocation with low overhead, albeit without memory isolation. The study also identifies current limitations and outlines potential research directions to improve temporal predictability in shared GPUs.

Figures (11)

• Figure 1: Conceptual diagram of the operation of a neural network ensemble system.
• Figure 2: Possible MIG instance configurations for the NVIDIA A100 GPU. Each color represents a possible MIG partition size based on the number of GPCs.
• Figure 3: Possible GC instance configurations for the NVIDIA Jetson Orin Nano GPU. Each color represents a possible GC partition size, based on the number of SMs and which SMs are utilized, depending on the assigned partition size.
• Figure 4: Figure of the results showing the effect of different memory sizes and compute units on the NVIDIA A100 using MIG for the inference of the ConvNeXt-Large network with a batch size of 1, illustrating the impact in terms of throughput, memory usage, and average power consumption.
• Figure 5: Figure of the results showing the effect of different SM assignments on the Jetson Orin Nano using GC for the inference of the ConvNeXt-Large network optimized, with a batch size of 1, illustrating the impact in terms of throughput, memory usage, and average power consumption.