ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design

TL;DR

ChipBench addresses the gap between academic benchmarks and industrial chip design workflows by introducing a comprehensive, three-task evaluation framework for AI-aided chip design: Verilog generation, Verilog debugging, and reference-model generation. It assembles 264 test cases across 44 generation modules, 89 debugging cases, and 132 reference-model samples in Python, SystemC, and CXXRTL, and provides an automated toolbox for training-data generation and verification. Empirical results reveal substantial performance gaps for current LLMs, with state-of-the-art models achieving only fractions of prior benchmark performance (e.g., ~30% on Verilog generation and ~13% on Python reference-model generation), underscoring the need for hierarchical-generation, waveform-aware debugging, and hardware-behavior modeling. The work also delivers an automated toolbox for generating high-quality reference-model training data, enabling scalable future research and offering practical support for industry adoption of AI-aided chip design workflows.

Abstract

While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the-art Claude-4.5-opus achieving only 30.74\% on Verilog generation and 13.33\% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95\% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at https://github.com/zhongkaiyu/ChipBench.git.

Figures (6)

• Figure 1: Overview of our ChipBench
• Figure 2: Demonstration of four types of bugs injected into original golden modules.
• Figure 3: Workflow of ChipBench across different tasks: (a) evaluating LLM-generated Verilog code, (b) assessing LLM-generated reference models (supporting Python, SystemC, and CXXRTL), and (c) utilizing the toolbox to generate high quality training datasets for reference models (supporting Python, SystemC, and CXXRTL, with Python shown as an example)
• Figure 4: Comparison of reset signal patterns between our approach and VerilogEval. We toggle reset once and only compare outputs during the normal operating phase to better reflect real-world verification standards and eliminate false negatives.
• Figure 5: Performance comparison between Zero-Shot debug tasks and One-Shot debug tasks. In Zero-Shot tests, LLMs are only aware of the buggy code and the existence of bugs, while in One-Shot tests, LLMs have access to the complete waveform file.