Optimal Kernel Tuning Parameter Prediction using Deep Sequence Models
Khawir Mahmood, Jehandad Khan, Hammad Afzal
TL;DR
The paper addresses the pressing challenge of tuning GPU kernel parameters within a vast, discrete search space by reframing it as a sequence-to-sequence translation problem. It develops a Seq2Seq framework with encoder–decoder structures, attention, and a convolutional hybrid encoder to map input tensor descriptors to sequences of kernel tuning parameters, augmented by constrained beam search to enforce hardware feasibility. Empirical results on AMD's MIOpen kernels show the approach achieving high prediction accuracy (often >95% for many kernels) and outperforming classical ML baselines, with the Hybrid-2 model and constraint techniques delivering the best results. The work demonstrates that learned kernel behavior can generalize to unseen problem configurations, promising reduced development time and compute resources for kernel tuning in production settings.
Abstract
GPU kernels have come to the forefront of computing due to their utility in varied fields, from high-performance computing to machine learning. A typical GPU compute kernel is invoked millions, if not billions of times in a typical application, which makes their performance highly critical. Due to the unknown nature of the optimization surface, an exhaustive search is required to discover the global optimum, which is infeasible due to the possible exponential number of parameter combinations. In this work, we propose a methodology that uses deep sequence-to-sequence models to predict the optimal tuning parameters governing compute kernels. This work considers the prediction of kernel parameters as a sequence to the sequence translation problem, borrowing models from the Natural Language Processing (NLP) domain. Parameters describing the input, output and weight tensors are considered as the input language to the model that emits the corresponding kernel parameters. In essence, the model translates the problem parameter language to kernel parameter language. The core contributions of this work are: a) Proposing that a sequence to sequence model can accurately learn the performance dynamics of a GPU compute kernel b) A novel network architecture which predicts the kernel tuning parameters for GPU kernels, c) A constrained beam search which incorporates the physical limits of the GPU hardware as well as other expert knowledge reducing the search space. The proposed algorithm can achieve more than 90% accuracy on various convolutional kernels in MIOpen, the AMD machine learning primitives library. As a result, the proposed technique can reduce the development time and compute resources required to tune unseen input configurations, resulting in shorter development cycles, reduced development costs, and better user experience.