ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints

TL;DR

ActionReasoningBench introduces a diagnostic benchmark for reasoning about actions and change (RAC), decomposing RAC into six dimensions and adding ramification constraints across eight IPC domains with action sequences up to 19 steps. The authors present a four-stage data-generation pipeline that translates PDDL domains to natural language, augments them with ramification constraints, and produces both binary and free-form questions evaluated under zero-shot and few-shot prompting for four LLMs, including GPT-4o and Llama variants. Key findings show LLMs perform reasonably on Fluent Tracking, State Tracking, and Effects of Actions, but struggle with Action Executability, Numerical RAC, and Composite Questions, with an average decline of about 17.9% on the more complex categories; ramification reasoning remains particularly challenging, as GPT-4o fails completely on ramification items while o1-preview achieves only about 18–19% on those questions. The benchmark exposes significant gaps in indirect-effect (ramification) reasoning and provides a scalable dataset and methodology to drive future improvements in RAC capabilities for LLMs, with implications for planning, commonsense reasoning, and interactive AI systems.

Abstract

Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems, such as the frame problem. It has driven advancements in AI fields, such as non-monotonic and commonsense reasoning. RAC remains crucial for AI systems that operate in dynamic environments, engage in interactive scenarios, or rely on commonsense reasoning. Despite substantial advances made by Large Language Models (LLMs) in various AI domains, their performance in RAC remains underexplored. To address this gap, we introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences. This benchmark rigorously evaluates LLMs across six key RAC dimensions: Fluent Tracking, State Tracking, Action Executability, Effects of Actions, Numerical RAC, and Composite Questions. LLMs demonstrate average accuracy rates of 73.55%, 65.63%, 58.73%, and 62.38% on the former four dimensions, which are frequently discussed in RAC literature. However, the performance on the latter two dimensions, which introduce complex and novel reasoning questions, the average performance of LLMs is lowered to 33.16% and 51.19%, respectively, reflecting a 17.9% performance decline. We also introduce new ramification constraints to capture the indirect effects of actions, providing deeper insights into RAC challenges. Our evaluation of state-of-the-art LLMs, including both open-source and commercial models, reveals challenges across all RAC dimensions, particularly in handling ramifications, with GPT-4o failing to solve any question and o1-preview achieving a score of only 18.4%.

Figures (3)

• Figure 1: Overview of the question generation pipeline for ActionReasoningBench. Blue blocks represent "Generated Data", and green blocks represent "Code used in the pipeline". Stage 1 involves generating states and plans using helmert2006fast and validating them with howey2004val. In Stage 2, PDDL instances and plans are converted to ASP. Stage 3 computes the action-state space through ASP. Stage 4 generates questions using templates, which are then rephrased to natural language via Llama-3.1-70B-Instruct.
• Figure 2: Performance for every Fluent Category for both binary and free-answer questions for every Action-Sequence length for GPT-4o, Llama-3.1-70B-Instruct, and Llama-3.1-8B-Instruct for Zero-shot-CoT prompt. Note: Bars represent SEM.
• Figure 3: Performance for every QUestion Category for both binary and free-answer questions for every Action-Sequence length for GPT-4o, Llama-3.1-70B-Instruct, and Llama-3.1-8B-Instruct for Zero-shot-CoT prompt. Note: Bars represent SEM.