Accurate GPU Memory Prediction for Deep Learning Jobs through Dynamic Analysis

TL;DR

This paper presents VeritasEst, a CPU-based tool that accurately predicts peak GPU memory for DL training without accessing GPUs, reducing OOM risk and enabling offline task scheduling. Grounded in observations about memory allocation, allocator behavior, and the sequence of memory activities, the method combines data analysis, data linking, and sequence orchestration to generate memory traces that approximate GPU usage. Through extensive experiments against static and DL-based baselines, VeritasEst achieves notable reductions in relative error and failure probability while delivering meaningful memory savings and reasonable runtime, demonstrating practical impact for GPU-constrained clusters. The approach offers a pathway to GPU-free memory prediction that can improve scheduling, resource utilization, and cost efficiency in real-world DL workloads.

Abstract

The benefits of Deep Learning (DL) impose significant pressure on GPU resources, particularly within GPU cluster, where Out-Of-Memory (OOM) errors present a primary impediment to model training and efficient resource utilization. Conventional OOM estimation techniques, relying either on static graph analysis or direct GPU memory profiling, suffer from inherent limitations: static analysis often fails to capture model dynamics, whereas GPU-based profiling intensifies contention for scarce GPU resources. To overcome these constraints, VeritasEst emerges. It is an innovative, entirely CPU-based analysis tool capable of accurately predicting the peak GPU memory required for DL training tasks without accessing the target GPU. This "offline" prediction capability is core advantage of VeritasEst, allowing accurate memory footprint information to be obtained before task scheduling, thereby effectively preventing OOM and optimizing GPU allocation. Its performance was validated through thousands of experimental runs across convolutional neural network (CNN) models: Compared to baseline GPU memory estimators, VeritasEst significantly reduces the relative error by 84% and lowers the estimation failure probability by 73%. VeritasEst represents a key step towards efficient and predictable DL training in resource-constrained environments.

Figures (9)

• Figure 1: Segment and Tensor memory usage changes are greatly influenced by where the zeroing gradient function, optimizer.zero_grad, is placed. Position 0 (POS0) means calling it just before backward propagation, while Position 1 (POS1) means calling it at the beginning of each iteration.
• Figure 2: Memory Management Diagram, showing how memory blocks are managed in PyTorch for a training job.
• Figure 3: The impact on GPU memory consumption by memory sequence activities
• Figure 4: Workflow of VeritasEst Memory Estimator.
• Figure 5: A call hierarchical structure diagram showing links between memory blocks, operators, and layers.