Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs

Yechen Xu, Yifei Wang, Nathanael Ren, Yiran Chen, Danyang Zhuo

TL;DR

Nixie tackles memory oversubscription on consumer GPUs by introducing a transparent temporal multiplexing system that coordinates GPU memory and kernel dispatch without requiring application or driver changes. It combines a chunk-based, hierarchical memory model with a lightweight LD_PRELOAD-enabled shim (Nixie Shim) and a centralized scheduler (Nixie Daemon) that jointly manage memory placement and compute, guided by an MLFQ-inspired policy with prefetching. Key contributions include the memory-planning and migration orchestrator, idleness-driven priority inference, and full CUDA-application compatibility, resulting in up to $3.8\times$ interactive latency reductions and up to $66.8\%$ less CPU pinned memory under equivalent latency. The approach demonstrates strong cross-workload improvements on real consumer GPUs and remains portable across platforms, highlighting practical impact for locally running large models and interactive AI pipelines.

Abstract

Consumer machines are increasingly running large ML workloads such as large language models (LLMs), text-to-image generation, and interactive image editing. Unlike datacenter GPUs, consumer GPUs serve single-user, rapidly changing workloads, and each model's working set often nearly fills the GPU memory. As a result, existing sharing mechanisms (e.g., NVIDIA Unified Virtual Memory) perform poorly due to memory thrashing and excessive use of CPU pinned memory when multiple applications are active. We design and implement Nixie, a system that enables efficient and transparent temporal multiplexing on consumer GPUs without requiring any application or driver changes. Nixie is a system service that coordinates GPU memory allocation and kernel launch behavior to efficiently utilize the CPU-GPU bi-directional bandwidth and CPU pinned memory. A lightweight scheduler in Nixie further improves responsiveness by automatically prioritizing latency-sensitive interactive jobs using MLFQ-inspired techniques. Our evaluations show that Nixie improves latency of real interactive code-completion tasks by up to $3.8\times$ and saves up to 66.8% CPU pinned memory usage given the same latency requirement.

Nixie: Efficient, Transparent Temporal Multiplexing for Consumer GPUs

TL;DR

Nixie tackles memory oversubscription on consumer GPUs by introducing a transparent temporal multiplexing system that coordinates GPU memory and kernel dispatch without requiring application or driver changes. It combines a chunk-based, hierarchical memory model with a lightweight LD_PRELOAD-enabled shim (Nixie Shim) and a centralized scheduler (Nixie Daemon) that jointly manage memory placement and compute, guided by an MLFQ-inspired policy with prefetching. Key contributions include the memory-planning and migration orchestrator, idleness-driven priority inference, and full CUDA-application compatibility, resulting in up to interactive latency reductions and up to less CPU pinned memory under equivalent latency. The approach demonstrates strong cross-workload improvements on real consumer GPUs and remains portable across platforms, highlighting practical impact for locally running large models and interactive AI pipelines.

Abstract

Consumer machines are increasingly running large ML workloads such as large language models (LLMs), text-to-image generation, and interactive image editing. Unlike datacenter GPUs, consumer GPUs serve single-user, rapidly changing workloads, and each model's working set often nearly fills the GPU memory. As a result, existing sharing mechanisms (e.g., NVIDIA Unified Virtual Memory) perform poorly due to memory thrashing and excessive use of CPU pinned memory when multiple applications are active. We design and implement Nixie, a system that enables efficient and transparent temporal multiplexing on consumer GPUs without requiring any application or driver changes. Nixie is a system service that coordinates GPU memory allocation and kernel launch behavior to efficiently utilize the CPU-GPU bi-directional bandwidth and CPU pinned memory. A lightweight scheduler in Nixie further improves responsiveness by automatically prioritizing latency-sensitive interactive jobs using MLFQ-inspired techniques. Our evaluations show that Nixie improves latency of real interactive code-completion tasks by up to and saves up to 66.8% CPU pinned memory usage given the same latency requirement.
Paper Structure (45 sections, 15 figures, 1 algorithm)

This paper contains 45 sections, 15 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Local ML Workloads on Consumer GPUs
  4. Limitations of Existing Multiplexing Approaches on Consumer GPUs
  5. Existing Approach #1: Shut Down an Application and Run the Next Application.
  6. Existing Approach #2: Application-Specific Approaches.
  7. Existing Approach #3: NVIDIA Unified Virtual Memory (UVM).
  8. Overview
  9. Supporting Unmodified CUDA Applications
  10. Maintaining consistent virtual memory addresses.
  11. Valid memory access during migration.
  12. CUDA graph compatibility.
  13. GPU Memory information.
  14. Virtual Memory Management
  15. Granularity of Memory Management
  16. ...and 30 more sections

Figures (15)

  • Figure 1: UVM's page fault driven migration. UVM can only utilize half link bandwidth for full-duplex PCIe channel.
  • Figure 2: Nixie's architecture overview. Green components belong to Nixie. Cubes with different colors stand for memory blocks from different applications.
  • Figure 3: Nixie's memory hierarchy and chunk migration.
  • Figure 4: Nixie's communication bandwidth usage (1) between GPU memory and CPU pinned memory, and (2) between CPU pinned memory to lower-tier memory (e.g., CPU paged memory, disk).
  • Figure 5: Migration plan given existing chunks' locations.
  • ...and 10 more figures