PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models

TL;DR

PCEval addresses the gap in evaluating large language models within hardware-dependent contexts by introducing a fully automated benchmark for physical computing. It decomposes tasks into four generation dimensions—logical circuit, physical circuit, code from logical circuits, and code from physical circuits—and validates outputs in a simulated Arduino environment to separate reasoning from implementation. The study shows LLMs perform comparatively well on code and logical circuit design but struggle with physically viable breadboard layouts, especially under higher complexity, with pin conflicts and breadboard-by-pass errors as dominant failure modes. Educational usability analyses and focus groups further reveal that AI tools must serve as scaffolding for learning, not just answer producers, demanding pedagogically aware interfaces and guidance. Overall, PCEval provides a scalable, automated foundation for advancing AI-assisted physical computing in classrooms and beyond, while outlining concrete paths to improve model understanding of physical constraints and practical hardware integration.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. Among these, software development is one of the key areas where LLMs are increasingly adopted. However, when hardware constraints are considered-for instance, in physical computing, where software must interact with and control physical hardware -their effectiveness has not been fully explored. To address this gap, we introduce \textsc{PCEval} (Physical Computing Evaluation), the first benchmark in physical computing that enables a fully automatic evaluation of the capabilities of LLM in both the logical and physical aspects of the projects, without requiring human assessment. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity. Through comprehensive testing of 13 leading models, \textsc{PCEval} provides the first reproducible and automatically validated empirical assessment of LLMs' ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. \textsc{PCEval} advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.

Figures (9)

• Figure 1: This figure illustrates the core protocols of the PCEval benchmark. The left panels show the input options for LLMs, while the right panels show the evaluation workflow. There are basically two types of tasks: code generation (blue arrow) and circuit generation (red arrow). Generated outputs undergo validation through standardized test procedures and circuit validation metrics, measuring both logical correctness and physical implementation feasibility in Arduino systems. (The circuit visualizations are shown to aid understanding; they are not used as inputs to the LLMs.)
• Figure 2: Impact of physical-constraint filters in D, C$\rightarrow$P: success rates are shown under progressively stricter criteria— blue: functional correctness only; orange: functional correctness and no breadboard bypass; green: functional correctness and no pin conflict; red: functional correctness and no breadboard bypass and no pin conflict. The drop from blue to red highlights how bypasses and pin conflicts drive failures in physical layout generation.
• Figure 3: Qualitative examples of LLM-generated physical circuits for a traffic light project (level 4), illustrating successful and failed attempts with corresponding error analyses.
• Figure 4: Qualitative examples of LLM-generated physical circuits for a servo motor basic project (level 1), illustrating successful and failed attempts with corresponding error analyses.
• Figure 5: Qualitative examples of LLM-generated physical circuits for a potentiometer servo motor project (level 3), illustrating successful and failed attempts with corresponding error analyses.