ProRAC: A Neuro-symbolic Method for Reasoning about Actions with LLM-based Progression

TL;DR

ProRAC introduces a three-step neuro-symbolic approach for RAC problems that uses LLM-driven preprocessing, progression, and querying to simulate action sequences and verify queries, without symbolic solvers. It demonstrates strong, cross-benchmark performance on TRAC, ACPBench, and ActionReasoningBench across multiple domains, with per-action progression helping manage long action sequences. The study connects observed errors to Frame, Ramification, and Qualification problems, highlighting enduring AI challenges in LLM-based reasoning and the need for robust benchmarks. Overall, ProRAC advances RAC research by showing that LLM-based progression can robustly handle action reasoning across diverse tasks, while pointing to future work in planning integrations and scalable prompting.

Abstract

In this paper, we propose ProRAC (Progression-based Reasoning about Actions and Change), a neuro-symbolic framework that leverages LLMs to tackle RAC problems. ProRAC extracts fundamental RAC elements including actions and questions from the problem, progressively executes each action to derive the final state, and then evaluates the query against the progressed state to arrive at an answer. We evaluate ProRAC on several RAC benchmarks, and the results demonstrate that our approach achieves strong performance across different benchmarks, domains, LLM backbones, and types of RAC tasks.

Figures (2)

• Figure 1: Overview of ProRAC, consisting of three steps: Preprocess, Progression and Querying.
• Figure 2: An illustrative example of ProRAC. The original data is processed step by step from Step One to Step Three.