Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentified Assessment of Logical Reasoning Agents

Zhiyu Ni, Yifeng Xiao, Zheng Liang

TL;DR

This work uses an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface.

Abstract

We present a framework for evaluating and benchmarking logical reasoning agents when assessment itself must be reproducible, auditable, and robust to execution failures. Building on agentified assessment, we use an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface. As a case study, we benchmark an auto-formalization agent for first-order logic (FOL) reasoning on a solver-verified and repaired split of FOLIO. The agent translates natural language premises and conclusions into executable Z3Py programs and employs satisfiability modulo theories (SMT) solving to determine logical entailment. On the cleaned FOLIO validation set, the auto-formalization agent achieves 86.70% accuracy under the assessor protocol, outperforming a chain-of-thought baseline (73.89%).

Agentified Assessment of Logical Reasoning Agents

TL;DR

This work uses an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface.

Abstract

We present a framework for evaluating and benchmarking logical reasoning agents when assessment itself must be reproducible, auditable, and robust to execution failures. Building on agentified assessment, we use an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types, while the agent under test only needs to expose a standardized agent-to-agent interface. As a case study, we benchmark an auto-formalization agent for first-order logic (FOL) reasoning on a solver-verified and repaired split of FOLIO. The agent translates natural language premises and conclusions into executable Z3Py programs and employs satisfiability modulo theories (SMT) solving to determine logical entailment. On the cleaned FOLIO validation set, the auto-formalization agent achieves 86.70% accuracy under the assessor protocol, outperforming a chain-of-thought baseline (73.89%).
Paper Structure (13 sections, 1 equation, 2 figures, 1 table)

This paper contains 13 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Benchmark and Data Cleaning
  3. Verification and Repair Pipeline
  4. Benchmark Statistics
  5. Agentified Evaluation Framework
  6. Agentified Assessment
  7. Assessor Agent (Evaluation Protocol)
  8. Reasoning Agents Under Test
  9. Implementation note.
  10. Experiments
  11. Experimental Setup and Baseline
  12. Results and Analysis
  13. Conclusion

Figures (2)

  • Figure 1: Overview of the data cleaning pipeline.
  • Figure 2: Traditional evaluation harnesses couple task execution, environments, and judging logic to a preset harness. In agentified assessment, an assessor agent evaluates an agent under test via an A2A interface a2a2026spec, reducing integration overhead.