Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

Hong Zhang, Barry Smith, Satish Balay, Le Chen, Murat Keceli, Lois Curfman McInnes, Junchao Zhang

Abstract

While large language models have significantly accelerated scientific code generation, comprehensively evaluating the generated code remains a major challenge. Traditional benchmarks reduce evaluation to test-case matching, an approach insufficient for library code in HPC where solver selection, API conventions, memory management, and performance are just as critical as functional correctness. To address this gap, we introduce petscagent-bench, an agentic framework built on an agents-evaluating-agents paradigm. Instead of relying on static scripts, petscagent-bench deploys a tool-augmented evaluator agent that compiles, executes, and measures code produced by a separate model-under-test agent, orchestrating a 14-evaluator pipeline across five scoring categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. Because the agents communicate through standardized protocols (A2A and MCP), the framework enables black-box evaluation of any coding agent without requiring access to its source code. We demonstrate the framework on a benchmark suite of realistic problems using the PETSc library for HPC. Our empirical analysis of frontier models reveals that while current models generate readable, well-structured code, they consistently struggle with library-specific conventions that traditional pass/fail metrics completely miss.

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

Abstract

While large language models have significantly accelerated scientific code generation, comprehensively evaluating the generated code remains a major challenge. Traditional benchmarks reduce evaluation to test-case matching, an approach insufficient for library code in HPC where solver selection, API conventions, memory management, and performance are just as critical as functional correctness. To address this gap, we introduce petscagent-bench, an agentic framework built on an agents-evaluating-agents paradigm. Instead of relying on static scripts, petscagent-bench deploys a tool-augmented evaluator agent that compiles, executes, and measures code produced by a separate model-under-test agent, orchestrating a 14-evaluator pipeline across five scoring categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. Because the agents communicate through standardized protocols (A2A and MCP), the framework enables black-box evaluation of any coding agent without requiring access to its source code. We demonstrate the framework on a benchmark suite of realistic problems using the PETSc library for HPC. Our empirical analysis of frontier models reveals that while current models generate readable, well-structured code, they consistently struggle with library-specific conventions that traditional pass/fail metrics completely miss.
Paper Structure (32 sections, 3 equations, 3 figures, 2 tables)

This paper contains 32 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Generation Benchmarks.
  4. HPC and Scientific Code Generation.
  5. Agent Architectures and Evaluation.
  6. An agentic evaluation framework for HPC libraries
  7. Design Principles
  8. Component Overview
  9. Protocol Usage
  10. Evaluation Pipeline
  11. Stage 1: Gates
  12. Stage 2: Metrics
  13. Numerical accuracy.
  14. Execution time.
  15. Stage 3: Quality Assessments
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Overview of the petscagent-bench architecture. The Green Agent (evaluator) communicates with the Purple Agent (model under test) through the A2A protocol and accesses compilation/execution tools through the MCP protocol.
  • Figure 2: Per-problem composite scores averaged over three runs per model.
  • Figure 3: Category-level scores.