Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agent Benchmarks Fail Public Sector Requirements

Jonathan Rystrøm, Chris Schmitz, Karolina Korgul, Jan Batzner, Chris Russell

TL;DR

Public-sector deployments of LLM-based agents risk compromising process legitimacy, transparency, and neutrality. The paper defines a theory-grounded rubric for PSO-relevant benchmarks—requiring process-based, realistic, public-sector-specific tasks with sector-specific metrics such as cost and fairness—and applies an LLM-assisted survey to over 1,300 benchmark papers. The analysis reveals that no existing benchmark satisfies all criteria, with major gaps in public-sector relevance and comprehensive metrics, especially fairness. The authors call for the development of public-sector benchmarks and for public-sector officials to evaluate benchmarks against the rubric to enable responsible agent adoption. Overall, the work highlights a critical misalignment between current benchmarking practices and the stringent needs of public administration, urging a shift toward ecologically valid evaluation for safe AI deployment in government contexts.

Abstract

Deploying Large Language Model-based agents (LLM agents) in the public sector requires assuring that they meet the stringent legal, procedural, and structural requirements of public-sector institutions. Practitioners and researchers often turn to benchmarks for such assessments. However, it remains unclear what criteria benchmarks must meet to ensure they adequately reflect public-sector requirements, or how many existing benchmarks do so. In this paper, we first define such criteria based on a first-principles survey of public administration literature: benchmarks must be \emph{process-based}, \emph{realistic}, \emph{public-sector-specific} and report \emph{metrics} that reflect the unique requirements of the public sector. We analyse more than 1,300 benchmark papers for these criteria using an expert-validated LLM-assisted pipeline. Our results show that no single benchmark meets all of the criteria. Our findings provide a call to action for both researchers to develop public sector-relevant benchmarks and for public-sector officials to apply these criteria when evaluating their own agentic use cases.

Agent Benchmarks Fail Public Sector Requirements

TL;DR

Public-sector deployments of LLM-based agents risk compromising process legitimacy, transparency, and neutrality. The paper defines a theory-grounded rubric for PSO-relevant benchmarks—requiring process-based, realistic, public-sector-specific tasks with sector-specific metrics such as cost and fairness—and applies an LLM-assisted survey to over 1,300 benchmark papers. The analysis reveals that no existing benchmark satisfies all criteria, with major gaps in public-sector relevance and comprehensive metrics, especially fairness. The authors call for the development of public-sector benchmarks and for public-sector officials to evaluate benchmarks against the rubric to enable responsible agent adoption. Overall, the work highlights a critical misalignment between current benchmarking practices and the stringent needs of public administration, urging a shift toward ecologically valid evaluation for safe AI deployment in government contexts.

Abstract

Deploying Large Language Model-based agents (LLM agents) in the public sector requires assuring that they meet the stringent legal, procedural, and structural requirements of public-sector institutions. Practitioners and researchers often turn to benchmarks for such assessments. However, it remains unclear what criteria benchmarks must meet to ensure they adequately reflect public-sector requirements, or how many existing benchmarks do so. In this paper, we first define such criteria based on a first-principles survey of public administration literature: benchmarks must be \emph{process-based}, \emph{realistic}, \emph{public-sector-specific} and report \emph{metrics} that reflect the unique requirements of the public sector. We analyse more than 1,300 benchmark papers for these criteria using an expert-validated LLM-assisted pipeline. Our results show that no single benchmark meets all of the criteria. Our findings provide a call to action for both researchers to develop public sector-relevant benchmarks and for public-sector officials to apply these criteria when evaluating their own agentic use cases.
Paper Structure (31 sections, 3 figures, 2 tables)

This paper contains 31 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Statement of Contributions:
  3. Background
  4. Agentic Large Language Models
  5. Defining LLM Agents:
  6. AI Agents and the Public Sector:
  7. Benchmarking
  8. Criteria for a Good Public-Sector Benchmark
  9. Benchmark Construction
  10. Task-based Model:
  11. Realistic Tasks:
  12. Public Sector-Specific Tasks:
  13. Reporting relevant metrics
  14. Cost:
  15. Fairness:
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Overview of process for formulation of public sector-relevant agent benchmark criteria.
  • Figure 2: Flow of how well the surveyed papers meet our criteria (see § \ref{['sec:criteria']}). No single paper meets all of the criteria, with the most challenging categories being public-sector relevance and relevant metrics.
  • Figure :