Automated Deep Learning Optimization via DSL-Based Source Code Transformation

TL;DR

Adopter introduces a domain-specific language to abstract DL model architectures and uses inter-procedural analysis, pattern matching, and synthesis-based code transformation to automatically integrate optimized DL kernels into source code. By representing model structures as DSL sequences and applying model transformation rules, Adopter achieves higher precision and recall in locating optimization opportunities than baseline approaches, while delivering tangible gains in training speed and GPU memory efficiency. The approach is validated on 199 Hugging Face models with nine optimization rules, showing substantial improvement in kernel integration and patch correctness, and is complemented by an open benchmark and publicly available code. The work advances practical DL optimization by reducing manual effort, enabling rapid deployment of kernel-level improvements with minimal runtime overhead, and outlining directions for broader kernel support and automatic rule inference.

Abstract

As deep learning models become increasingly bigger and more complex, it is critical to improve model training and inference efficiency. Though a variety of highly optimized libraries and packages (known as DL kernels) have been developed, it is tedious and time-consuming to figure out which kernel to use, where to use, and how to use them correctly. To address this challenge, we propose an Automated Deep learning OPTimization approach called Adopter. We design a Domain-Specific Language (DSL) to represent DL model architectures and leverage this DSL to specify model transformation rules required to integrate a DL kernel into a model. Given the source code of a DL model and the transformation rules for a set of kernels, Adopter first performs inter-procedural analysis to identify and express the model architecture in our DSL. Then, Adopter performs scope analysis and sub-sequence matching to identify locations in the model architecture where the transformation rules can be applied. Finally, Adopter proposes a synthesis-based code transformation method to apply the transformation rule. We curated a benchmark with 199 models from Hugging Face and a diverse set of DL kernels. We found that, compared to a state-of-the-art automated code transformation technique, Adopter helps improve the precision and recall by 3% and 56%, respectively. An in-depth analysis of 9 models revealed that on average, Adopter improved the training speed by 22.7% while decreasing the GPU memory usage by 10.5%.

Figures (6)

• Figure 1: The code transformation performed by Adopter to fuse Dropout and LayerNorm layers using the EPOI kernel
• Figure 2: Domain-Specific Language
• Figure 3: Code snippets from Hugging Face DPT model and the two possible model structures extracted from it.
• Figure 4: Replacing two statements in different basic blocks. (a) $\bm{N_3}$ lies in a branch while $\bm{N_2}$ does not; (b) Adopter makes two copies of $\bm{N_2}$ into the two branches; (c) Adopter replaces copied $\bm{N_2}$ and $\bm{N_3}$ with a new statement $\bm{N_{1}'}$
• Figure 5: Transforming model DPT using Adopter.