LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actions

TL;DR

The paper tackles the challenge of complex action reasoning by marrying LLMs with the expressive, semantically grounded $BC^+$ action language. It proposes a four-stage LLM+AL pipeline that generates a BC^+ signature, extracts knowledge, constructs causal rules, and iteratively revises the program using solver feedback. On Missionaries and Cannibals elaborations, the approach achieves correct plans with only a few human corrections, outperforming several strong LLM baselines that struggle with elaborations and state constraints. The Self-Revision loop dramatically improves executability and correctness, demonstrating a path toward robust, automated symbolic reasoning atop natural language processing.

Abstract

Large Language Models (LLMs) have made significant strides in various intelligent tasks but still struggle with complex action reasoning tasks that require systematic search. To address this limitation, we propose a method that bridges the natural language understanding capabilities of LLMs with the symbolic reasoning strengths of action languages. Our approach, termed "LLM+AL," leverages the LLM's strengths in semantic parsing and commonsense knowledge generation alongside the action language's proficiency in automated reasoning based on encoded knowledge. We compare LLM+AL against state-of-the-art LLMs, including ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, and o1-preview, using benchmarks for complex reasoning about actions. Our findings indicate that, although all methods exhibit errors, LLM+AL, with relatively minimal human corrections, consistently leads to correct answers, whereas standalone LLMs fail to improve even with human feedback. LLM+AL also contributes to automated generation of action languages.

Figures (2)

• Figure 1: LLM+AL pipeline
• Figure 2: LLM+AL