Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ASP-Bench: From Natural Language to Logic Programs

Stefan Szeider

TL;DR

ASP-Bench proposes a semantically verified, NL-to-ASP benchmark suite with 128 problem instances to systematically cover ASP features and reasoning patterns. It demonstrates an agentic, feedback-driven approach based on the ReAct framework that achieves full saturation on easy and hard problems, highlighting the value of solver feedback for iterative ASP modeling. The work also contrasts MCP-based integration strategies, analyzes problem hardness beyond feature counts, and discusses practical implications for neurosymbolic engineering, benchmark design, and future NL-to-ASP research. Overall, the benchmark provides a rigorous, extensible platform for evaluating and guiding NL-to-ASP systems and contributes insights into modeling strategies, efficiency trade-offs, and solver-driven debugging.

Abstract

Automating the translation of natural-language specifications into logic programs is a challenging task that affects neurosymbolic engineering. We present ASP-Bench, a benchmark comprising 128 natural language problem instances, 64 base problems with easy and hard variants. It evaluates systems that translate natural-language problems into Answer Set Programs (ASPs), a prominent form of logic programming. It provides systematic coverage of ASP features, including choice rules, aggregates, and optimization. Each problem includes reference validators that check whether solutions satisfy the problem specification. We characterize problems along seven largely independent reasoning aspects (optimization, temporal reasoning, default logic, resource allocation, recursion, spatial reasoning, and quantitative complexity), providing a multidimensional view of modeling difficulty. We test the benchmark using an agentic approach based on the ReAct (Reason and Act) framework, which achieves full saturation, demonstrating that feedback-driven iterative refinement with solver feedback provides a reliable and robust approach for modeling natural language in ASP. Our analysis across multiple agent runs enables us to gain insights into what determines a problem's modeling hardness.

ASP-Bench: From Natural Language to Logic Programs

TL;DR

ASP-Bench proposes a semantically verified, NL-to-ASP benchmark suite with 128 problem instances to systematically cover ASP features and reasoning patterns. It demonstrates an agentic, feedback-driven approach based on the ReAct framework that achieves full saturation on easy and hard problems, highlighting the value of solver feedback for iterative ASP modeling. The work also contrasts MCP-based integration strategies, analyzes problem hardness beyond feature counts, and discusses practical implications for neurosymbolic engineering, benchmark design, and future NL-to-ASP research. Overall, the benchmark provides a rigorous, extensible platform for evaluating and guiding NL-to-ASP systems and contributes insights into modeling strategies, efficiency trade-offs, and solver-driven debugging.

Abstract

Automating the translation of natural-language specifications into logic programs is a challenging task that affects neurosymbolic engineering. We present ASP-Bench, a benchmark comprising 128 natural language problem instances, 64 base problems with easy and hard variants. It evaluates systems that translate natural-language problems into Answer Set Programs (ASPs), a prominent form of logic programming. It provides systematic coverage of ASP features, including choice rules, aggregates, and optimization. Each problem includes reference validators that check whether solutions satisfy the problem specification. We characterize problems along seven largely independent reasoning aspects (optimization, temporal reasoning, default logic, resource allocation, recursion, spatial reasoning, and quantitative complexity), providing a multidimensional view of modeling difficulty. We test the benchmark using an agentic approach based on the ReAct (Reason and Act) framework, which achieves full saturation, demonstrating that feedback-driven iterative refinement with solver feedback provides a reliable and robust approach for modeling natural language in ASP. Our analysis across multiple agent runs enables us to gain insights into what determines a problem's modeling hardness.
Paper Structure (21 sections, 2 figures, 2 tables)

This paper contains 21 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Example: Knights and Knaves
  3. The ASP-Bench Benchmark
  4. Problem Construction Process
  5. Semantic Verification
  6. Problem Characterization
  7. ASP Agent
  8. Experimental Evaluation
  9. Experimental Setup
  10. Saturation on Easy and Hard Problems
  11. Problem Hardness and Reasoning Aspects
  12. Agent Activity Patterns
  13. Anatomy of Agentic Modeling
  14. Problem Complexity: The Pilgrim's Journey
  15. The Expert Strategy (6 Calls)
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Top: Average python_exec calls per problem (hardness measure). Bottom: Reasoning aspects per problem. Problems are ordered by hardness (ascending).
  • Figure 2: Left: Distribution of agent activity types across difficulty ranges (1--10, 11--20, 21--30, 31+ executions). Right: Agent behavior metrics showing time per execution (blue, left axis) and input/output token ratio (orange, right axis).