HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks

TL;DR

High-level synthesis (HLS) design increasingly leverages LLMs, but existing benchmarks focus on HDL and lack a holistic, extensible framework for HLS tasks. The paper presents HLS-Eval, a 94-design benchmark plus an open-source Python framework that enables end-to-end, parallel evaluation of LLMs on HLS code generation and hardware-optimizing code editing. It operationalizes evaluation with parseability, compilability, runnability, and synthesizability metrics and reports unbiased pass@k baselines across several open-source models, using integrated Vitis HLS tooling. By open-sourcing the benchmark and framework, the work aims to accelerate research in LLM-assisted HLS design and facilitate future tool-feedback evaluators and richer benchmarks.

Abstract

The rapid scaling of large language model (LLM) training and inference has driven their adoption in semiconductor design across academia and industry. While most prior work evaluates LLMs on hardware description language (HDL) tasks, particularly Verilog, designers are increasingly using high-level synthesis (HLS) to build domain-specific accelerators and complex hardware systems. However, benchmarks and tooling to comprehensively evaluate LLMs for HLS design tasks remain scarce. To address this, we introduce HLS-Eval, the first complete benchmark and evaluation framework for LLM-driven HLS design. HLS-Eval targets two core tasks: (1) generating HLS code from natural language descriptions, and (2) performing HLS-specific code edits to optimize performance and hardware efficiency. The benchmark includes 94 unique designs drawn from standard HLS benchmarks and novel sources. Each case is prepared via a semi-automated flow that produces a natural language description and a paired testbench for C-simulation and synthesis validation, ensuring each task is "LLM-ready." Beyond the benchmark, HLS-Eval offers a modular Python framework for automated, parallel evaluation of both local and hosted LLMs. It includes a parallel evaluation engine, direct HLS tool integration, and abstractions for to support different LLM interaction paradigms, enabling rapid prototyping of new benchmarks, tasks, and LLM methods. We demonstrate HLS-Eval through baseline evaluations of open-source LLMs on Vitis HLS, measuring outputs across four key metrics - parseability, compilability, runnability, and synthesizability - reflecting the iterative HLS design cycle. We also report pass@k metrics, establishing clear baselines and reusable infrastructure for the broader LLM-for-hardware community. All benchmarks, framework code, and results are open-sourced at https://github.com/stefanpie/hls-eval.

Figures (7)

• Figure 1: Overview of HLS-Eval, which includes benchmark construction, an evaluation framework, and HLS design tasks such as code generation and optimization-based code editing evaluations.
• Figure 2: Overview of the LLM-aided benchmark construction given some arbitrary unstructured HLS source code. Note the human-in-the-loop for manually tweaking and review of the final benchmark design.
• Figure 3: The parallel benchmark evaluation backend for HLS-Eval. Execution at each stage of the deigns flow can be submitted to a fixed-sized pool of threads on a server or user workstation. Different stages can have different sized pools to set different parallelisms for each stage to maximize utilization of compute resources
• Figure 4: An execution trace of the HLS-Eval parallel benchmark elevation backend. The graphs show the pool utilization of all tasks pools over time. The red line shows the user set pool size for each task.
• Figure 5: Evaluation results of HLS code generation and HLS code editing tasks; "Pass Rate": average pass@k over all evaluated designs for a given design stage and model