Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation

TL;DR

Bench4HLS addresses the need for rigorous, end-to-end benchmarking of LLM-based high-level synthesis by introducing a diverse 170-design dataset, a pluggable PPA analysis interface, and a four-stage evaluation pipeline that spans code generation, compilation, verification, and post-RTL synthesis. The framework enables design-space exploration through YAML-driven directives and cross-tool evaluation (Vitis HLS and Vivado) to quantify functional correctness and hardware metrics across multiple LLMs. Experimental results demonstrate that larger models like GPT-5 achieve higher synthesis and verification success rates, and that YAML-based DSE can yield meaningful PPA improvements in a significant fraction of cases, while reference designs remain strong baselines for PPA efficiency. Overall, Bench4HLS provides a reproducible, hardware-aware benchmark that can drive fair comparisons and guide future enhancements in LLM-assisted HLS workflows.

Abstract

In last two years, large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL). While their use in high-level synthesis (HLS) remains comparatively less mature, the ratio of HLS- to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months, indicating growing interest in leveraging LLMs for high-level design entry while relying on downstream synthesis for optimization. This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS. To address this, We present Bench4HLS for evaluating LLM-generated HLS designs. Bench4HLS comprises 170 manually drafted and validated case studies, spanning small kernels to complex accelerators, curated from widely used public repositories. The framework supports fully automated assessment of compilation success, functional correctness via simulation, and synthesis feasibility/optimization. Crucially, Bench4HLS integrates a pluggable API for power, performance, and area (PPA) analysis across various HLS toolchains and architectures, demonstrated here with Xilinx Vitis HLS and validated on Catapult HLS. By providing a structured, extensible, and plug-and-play testbed, Bench4HLS establishes a foundational methodology for benchmarking LLMs in HLS workflows.

Figures (4)

• Figure 1: Overview of Bench4HLS: The whole workflow of the evaluation method
• Figure 2: PPA Analysis of LLM-generated (GPT) HLS Codes by Bench4HLS. Bench4HLS reports percentage differences between the generated designs by MUT and the reference designs provided by Bench4HLS across four metrics (LUT utilization, FF utilization, post-synthesis power, and latency -- for pass@1, pass@5, and pass@10 generations), with values representing (generated – reference)%. Positive means higher resource usage or latency vs. the reference.
• Figure 3: HLS Code Generation Evaluated by Bench4HLS for Pass@10 (DSE refers to improvement of at least one of PPA metrics less/more than 20%).
• Figure 4: Power Consumption Benchmarking in MUTs Tested by Bench4HLS.