ChiBench: a Benchmark Suite for Testing Electronic Design Automation Tools

TL;DR

ChiBench addresses the shortage of large HDL benchmarks by delivering a Verilog-focused suite of about 50K programs mined from public GitHub repositories and filtered with Verible for syntax and Jasper for semantics. The paper demonstrates the suite’s utility across empirical complexity analysis, coverage assessment, and bug discovery in EDA tools, revealing linear parsing behavior with respect to token count and modest but informative coverage metrics. It also reports actual bugs found in Verible components and discusses how ChiBench can seed new program generation via probabilistic grammars for broader tool testing. The work implies significant practical impact for stress-testing EDA workflows and offers a foundation for expanding benchmark coverage to other HDL languages like VHDL, with open-source availability to the community.

Abstract

Electronic Design Automation (EDA) tools are software applications used by engineers in the design, development, simulation, and verification of electronic systems and integrated circuits. These tools typically process specifications written in a Hardware Description Language (HDL), such as Verilog, SystemVerilog or VHDL. Thus, effective testing of these tools requires benchmark suites written in these languages. However, while there exist some open benchmark suites for these languages, they tend to consist of only a handful of specifications. This paper, in contrast, presents ChiBench, a comprehensive suite comprising 50 thousand Verilog programs. These programs were sourced from GitHub repositories and curated using Verible's syntactic analyzer and Jasper(TM)'s HDL semantic analyzer. Since its inception, ChiBench has already revealed bugs in public tools like Verible's obfuscator and parser. In addition to explaining some of these case studies, this paper demonstrates how ChiBench can be used to evaluate the asymptotic complexity and code coverage of typical electronic design automation tools.

Figures (6)

• Figure 1: Verilog specification filtered out by our syntactic verification.
• Figure 2: Verilog specification that fails the semantic test.
• Figure 3: Licenses of repositories used to build ChiBench.
• Figure 4: The impact of the number of tokens on Verible's parsing time. This figure contains about 50K points---one point per program in the ChiBench collection.
• Figure 5: Code coverage of different tools of the Verible Framework, using the 128 largest programs from ChiBench.