LIFT: LLM-Based Pragma Insertion for HLS via GNN Supervised Fine-Tuning

TL;DR

LIFT tackles the challenge of automatic pragma insertion in HLS by marrying an LLM with graph-based structural supervision to capture the semantic and microarchitectural effects of pragmas. The method uses Infilling by Language Modeling to predict missing pragma factors, and supervises the LLM with a lightweight GNN via ProGraML-encoded program graphs, combining a token-level loss with a graph-embedding loss and latency-aware weighting. Training on the HLSyn dataset with intelligent resampling and a 3-epoch workflow on 8 GPUs yields large QoR gains, achieving up to $3.52\times$ faster latency than AutoDSE-24h, $2.16\times$ faster than HARP, and $66\times$ faster than GPT-4o, while maintaining high validity across designs. LIFT also demonstrates robustness to tool-version shifts, outperforming prior GNN-based approaches by about $2.74\times$ under Vitis 2021.1, indicating strong generalization. Overall, LIFT advances LLM-guided hardware design by integrating structural semantics, failure-aware learning, and latency optimization into a single, scalable framework.

Abstract

FPGAs are increasingly adopted in datacenter environments for their reconfigurability and energy efficiency. High-Level Synthesis (HLS) tools have eased FPGA programming by raising the abstraction level from RTL to untimed C/C++, yet attaining high performance still demands expert knowledge and iterative manual insertion of optimization pragmas to modify the microarchitecture. To address this challenge, we propose LIFT, a large language model (LLM)-based coding assistant for HLS that automatically generates performance-critical pragmas given a C/C++ design. We fine-tune the LLM by tightly integrating and supervising the training process with a graph neural network (GNN), combining the sequential modeling capabilities of LLMs with the structural and semantic understanding of GNNs necessary for reasoning over code and its control/data dependencies. On average, LIFT produces designs that improve performance by 3.52x and 2.16x than prior state-of the art AutoDSE and HARP respectively, and 66x than GPT-4o.

Figures (5)

• Figure 1: Two data points from the HLSyn chang2023dr database for the GEMM-B kernel.
• Figure 2: Our goal is to develop an LLM-based coding assistant which can take as input an HLS code without the optimization pragmas, and get as output the optimal pragma configuration with the least latency.
• Figure 3: Training loss for the Deepseek Coder model along multiple training steps. The stagnation indicates that the model is unable to grasp the impact of different pragmas.
• Figure 4: TSNE visualization of 1000 representative points from the training database embedded using the Deepseek Coder model. Despite the vastly different pragma annotations for each design point, the LLM clusters the different data points from the same kernel in tight clusters, indicating that the model cannot understand pragmas despite fine-tuning. This motivates us to supervise the LLM training with the structural and semantic understanding of a GNN.
• Figure 5: Overview of the LIFT training framework. (a) The LLM runs a forward pass on the input training data. (b) The predicted pragmas and target ground truth pragmas are inserted back into the training code to obtain compilable versions. These compilable C codes are converted into graphs and passed through the HARP GNN. (c) The embedding loss between the graphs is combined with the autoregressive loss. (d) The combined loss is used to back-propagate and update the LLM.