CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs

TL;DR

The paper introduces CIRCUIT, a benchmark designed to probe Large Language Models for analog circuit interpretation and multi-level reasoning across circuit topologies. It assembles 510 questions from 102 templates with five numerical setups each, including diagrams and adapted netlists to test topology understanding, and evaluates models with unit-test–like pass@k/n metrics alongside global accuracy. The findings show that GPT-4o is the most robust among tested models, particularly when netlists and limited-one-shot exemplars are provided, yet performance remains limited in generalizing across varied numerical setups and complex topologies, underscoring substantial challenges in circuit reasoning. The work demonstrates the value of unit-test–style evaluation for diagnosing reasoning gaps and suggests directions like integrating simulators or interpreters, expanding topologies, and applying the framework to broader reasoning domains. Overall, CIRCUIT offers a cost-effective, scalable, and transparent pathway to advance LLM-assisted analog circuit design by highlighting current capabilities and concrete avenues for improvement.

Abstract

The role of Large Language Models (LLMs) has not been extensively explored in analog circuit design, which could benefit from a reasoning-based approach that transcends traditional optimization techniques. In particular, despite their growing relevance, there are no benchmarks to assess LLMs' reasoning capability about circuits. Therefore, we created the CIRCUIT dataset consisting of 510 question-answer pairs spanning various levels of analog-circuit-related subjects. The best-performing model on our dataset, GPT-4o, achieves 48.04% accuracy when evaluated on the final numerical answer. To evaluate the robustness of LLMs on our dataset, we introduced a unique feature that enables unit-test-like evaluation by grouping questions into unit tests. In this case, GPT-4o can only pass 27.45% of the unit tests, highlighting that the most advanced LLMs still struggle with understanding circuits, which requires multi-level reasoning, particularly when involving circuit topologies. This circuit-specific benchmark highlights LLMs' limitations, offering valuable insights for advancing their application in analog integrated circuit design.

Figures (4)

• Figure 1: A simplified overview of the CIRCUIT dataset and experiment setup. Analog circuit problems, sourced from various materials, are adapted into the CIRCUIT dataset, comprising 510 problems. We assess the performance of three Large Language Models (GPT 4o, GPT Turbo, Gemini 1.5 Pro) in understanding analog circuits and their topologies from diagrams and netlists, using four distinct prompt designs. The LLMs' responses are then evaluated both automatically and manually, with unique evaluation metrics designed to reveal higher-level insights and capture the effects of data homogeneity. Quantitative analysis and human error analysis were done to assess model performance in reasoning about analog circuits.
• Figure 2: Example datapoint from the CIRCUIT dataset. Each datapoint includes a template question, which may or may not have an associated diagram. In most cases, diagrams are further supplemented by netlists that describe the circuit's components and connections. Additionally, each datapoint is associated with a unique numerical setup.
• Figure 3: Templates distribution across categories and levels. The heatmap displays the distribution of templates in the CIRCUIT dataset across four categories (Analog, Basic, Power, and Radio Frequency) and three levels (1, 3, and 5). The numbers inside each cell represent the total count of templates, with percentages indicating the proportion of templates relative to the entire dataset (totaling 102 templates). The color intensity corresponds to the number of templates, as indicated by the color bar on the right.
• Figure :