ForgeBench: A Machine Learning Benchmark Suite and Auto-Generation Framework for Next-Generation HLS Tools

TL;DR

ForgeBench addresses two core barriers to HLS adoption by delivering an ML-focused benchmark suite and an auto-generation framework that supports architecture-oriented design and modular reuse. It provides over 6,000 ML-oriented designs and a modular HLS suite that emphasizes shared computational blocks across GEMM, Convolution, and attention modules, enabling evaluation of future HLS tools capable of automatic module extraction. The framework uses JSON-configured designs with a C/C++ template library, multi-threaded synthesis, and production of PPA reports, currently targeting Xilinx Vitis HLS but extensible to other flows. The results demonstrate extensive reuse opportunities across core ML operators and models, underscoring the practical impact of architecture-aware HLS tooling. ForgeBench is open-source and extensible, offering a scalable platform to advance next-generation HLS tool design and evaluation.

Abstract

Although High-Level Synthesis (HLS) has attracted considerable interest in hardware design, it has not yet become mainstream due to two primary challenges. First, current HLS hardware design benchmarks are outdated as they do not cover modern machine learning (ML) applications, preventing the rigorous development of HLS tools on ML-focused hardware design. Second, existing HLS tools are outdated because they predominantly target individual accelerator designs and lack an architecture-oriented perspective to support common hardware module extraction and reuse, limiting their adaptability and broader applicability. Motivated by these two limitations, we propose ForgeBench, an ML-focused benchmark suite with a hardware design auto-generation framework for next-generation HLS tools. In addition to the auto-generation framework, we provide two ready-to-use benchmark suites. The first contains over 6,000 representative ML HLS designs. We envision future HLS tools being architecture-oriented, capable of automatically identifying common computational modules across designs, and supporting flexible dataflow and control. Accordingly, the second benchmark suite includes ML HLS designs with possible resource sharing manually implemented to highlight the necessity of architecture-oriented design, ensuring it is future-HLS ready. ForgeBench is open-sourced at https://github.com/hchen799/ForgeBench .

Figures (2)

• Figure 1: Proposed benchmarking framework enabling future HLS tools for architecture design.
• Figure 2: Resource/Latency trade-off with common GEMM tiles.