SIP: Autotuning GPU Native Schedules via Stochastic Instruction Perturbation
Guoliang He, Eiko Yoneki
TL;DR
The paper tackles optimizing GPU native instruction schedules to push CUDA kernel performance for transformer-based LLM workloads. It introduces SIP, an automatic optimizer that uses stochastic perturbations of the native SASS schedule to produce a better cubin. The approach defines a pruned search space focused on global memory I/O, a mutation policy, a runtime feedback signal, and a simulated annealing search loop, implemented within Triton via Cuasm. Experiments on two representative kernels show about a 6–12% latency/throughput improvement, with validation based on up to $10^7$ test samples, demonstrating the practicality of automatic native-schedule optimization for large-scale models. These results indicate that automated native-schedule optimization can complement hand-crafted CUDA kernels to extract additional performance for LLM workloads.
Abstract
Large language models (LLMs) have become a significant workload since their appearance. However, they are also computationally expensive as they have billions of parameters and are trained with massive amounts of data. Thus, recent works have developed dedicated CUDA kernels for LLM training and inference instead of relying on compilergenerated ones, so that hardware resources are as fully utilized as possible. In this work, we explore the possibility of GPU native instruction optimization to further push the CUDA kernels to extreme performance. Contrary to prior works, we adopt an automatic optimization approach by defining a search space of possible GPU native instruction schedules, and then we apply stochastic search to perform optimization. Experiments show that SIP can further improve CUDA kernel throughput by automatically discovering better GPU native instruction schedules and the optimized schedules are tested by 10 million test samples.