QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

TL;DR

QuArch introduces the first benchmark dedicated to evaluating LLM reasoning and knowledge in computer architecture, assembling 2,671 expert-validated QA pairs that span processor design, memory systems, and interconnects. It formalizes a four-skill framework—Recall, Analyze, Design, Implement—and uses a mixed-generation pipeline (synthetic, academic, and exam-derived questions) validated by both LLM-based judges and domain experts to measure performance across ten frontier models. The findings show strong recall across models but a substantial gap in higher-order architectural reasoning, with design and implementation tasks exhibiting the largest deficits and considerable model-to-model variation. By detailing failure modes, validating judgment alignment with human experts, and evaluating multimodal inputs, QuArch lays a foundation for agentic, reasoning-centered progress in computer architecture and provides a standardized benchmark for future AI-assisted computing system design.

Abstract

The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from current large language model (LLM) evaluations. To this end, we present QuArch (pronounced 'quark'), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. QuArch provides a comprehensive collection of 2,671 expert-validated question-answer (QA) pairs covering various aspects of computer architecture, including processor design, memory systems, and interconnection networks. Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking in computer architecture. Frontier model accuracies vary widely (from 34% to 72%) on these advanced questions, highlighting persistent gaps in architectural reasoning across analysis, design, and implementation QAs. By holistically assessing fundamental skills, QuArch provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. With over 140 contributors from 40 institutions, this benchmark represents a community effort to set the standard for architectural reasoning in LLM evaluation.

Figures (16)

• Figure 1: Reported results valsai_benchmarks for different models across QuArch and multiple other SoTA benchmarks to date.
• Figure 2: QuArch QA Skills Framework. The benchmark evaluates four complementary competencies: Recall, Analyze, Design, and Implement. QAs in QuArch contain relevant context describing the scenario, constraints, and figures when appropriate. The illustrative example shows how distinct question styles derived from the same context can probe different skills. Together, these competencies form the building blocks required for systems design in computer architecture. Balanced mastery of skills will enable more effective agents and workflows across the computing stack.
• Figure 3: We construct QuArch with a three-pronged approach including a blend of synthetic data generation, community crowdsourcing, and academic exams. All QAs are validated by a human expert to curate QuArch's final benchmark set of 2,671 question-answer pairs.
• Figure 4: Distribution of topics in QuArch.
• Figure 5: Breakdown of skill, format, and modality distributions in QuArch.