Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ML-Based Optimum Number of CUDA Streams for the GPU Implementation of the Tridiagonal Partition Method

Milena Veneva, Toshiyuki Imamura

TL;DR

The paper tackles determining the optimal number of CUDA streams for a GPU implementation of the tridiagonal partition method used to solve SLAEs. It develops a regression-based heuristic built on a refined two-stage GPU time model, incorporating sum-of-overlapped times and stream-creation overhead, and validates it with empirical data from multiple SLAE sizes on RTX GPUs. Key contributions include a linear model for the overlapped GPU time sum, two regime-based overhead models, and an algorithm that predicts the optimum stream count with high agreement to observed data, achieving up to 1.3× speedups for large problems. The work demonstrates FP32 adjustments and cross-GPU generalizability, offering a practical approach for optimizing GPU concurrency in AI-oriented HPC contexts.

Abstract

This paper presents a heuristic for finding the optimum number of CUDA streams by using tools common to the modern AI-oriented approaches and applied to the parallel partition algorithm. A time complexity model for the GPU realization of the partition method is built. Further, a refined time complexity model for the partition algorithm being executed on multiple CUDA streams is formulated. Computational experiments for different SLAE sizes are conducted, and the optimum number of CUDA streams for each of them is found empirically. Based on the collected data a model for the sum of the times for the non-dominant GPU operations (that take part in the stream overlap) is formulated using regression analysis. A fitting non-linear model for the overhead time connected with the creation of CUDA streams is created. Statistical analysis is done for all the built models. An algorithm for finding the optimum number of CUDA streams is formulated. Using this algorithm, together with the two models mentioned above, predictions for the optimum number of CUDA streams are made. Comparing the predicted values with the actual data, the algorithm is deemed to be acceptably good.

ML-Based Optimum Number of CUDA Streams for the GPU Implementation of the Tridiagonal Partition Method

TL;DR

The paper tackles determining the optimal number of CUDA streams for a GPU implementation of the tridiagonal partition method used to solve SLAEs. It develops a regression-based heuristic built on a refined two-stage GPU time model, incorporating sum-of-overlapped times and stream-creation overhead, and validates it with empirical data from multiple SLAE sizes on RTX GPUs. Key contributions include a linear model for the overlapped GPU time sum, two regime-based overhead models, and an algorithm that predicts the optimum stream count with high agreement to observed data, achieving up to 1.3× speedups for large problems. The work demonstrates FP32 adjustments and cross-GPU generalizability, offering a practical approach for optimizing GPU concurrency in AI-oriented HPC contexts.

Abstract

This paper presents a heuristic for finding the optimum number of CUDA streams by using tools common to the modern AI-oriented approaches and applied to the parallel partition algorithm. A time complexity model for the GPU realization of the partition method is built. Further, a refined time complexity model for the partition algorithm being executed on multiple CUDA streams is formulated. Computational experiments for different SLAE sizes are conducted, and the optimum number of CUDA streams for each of them is found empirically. Based on the collected data a model for the sum of the times for the non-dominant GPU operations (that take part in the stream overlap) is formulated using regression analysis. A fitting non-linear model for the overhead time connected with the creation of CUDA streams is created. Statistical analysis is done for all the built models. An algorithm for finding the optimum number of CUDA streams is formulated. Using this algorithm, together with the two models mentioned above, predictions for the optimum number of CUDA streams are made. Comparing the predicted values with the actual data, the algorithm is deemed to be acceptably good.
Paper Structure (9 sections, 7 equations, 4 figures, 5 tables)

This paper contains 9 sections, 7 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Building a heuristic for the optimum number of CUDA streams
  3. Hardware working queues and CUDA streams.
  4. Time complexity model.
  5. Overview of existing heuristics.
  6. Building mathematical models for $\textrm{sum}$, and $T\mathrm{\_{overhead}}$.
  7. Discussion and Conclusions
  8. Experiments on other NVIDIA GPU cards.
  9. Experiments with FP32 precision.

Figures (4)

  • Figure 1: Nsight Systems profiles of the kernel responsible for Stage 1 (a), and the kernel responsible for Stage 3 (b); SLAE size $10^{8}$, sub-system size 10, 10 CUDA streams. Cyan/pink boxes denote H2D/D2H memory transfers; blue -- kernels.
  • Figure 2: Comparison of the sum of times for different SLAE sizes.
  • Figure 3: $T\mathrm{\_{overhead}}$ for different SLAE sizes, on different number of CUDA streams.
  • Figure 4: Distribution of the actual vs. predicted values for the model of $T\mathrm{\_{overhead}}$ for SLAE sizes $\leq10^{6}$ (a), and $>10^{6}$ (b).