Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts

TL;DR

A novel approach that constructs an action schema library to generate multiple candidates, accounting for the diverse possible interpretations of natural language descriptions, and introduces a semantic validation and ranking module that automatically filter and rank these candidates without expert-in-the-loop.

Abstract

Large Language Models (LLMs) have shown promise in solving natural language-described planning tasks, but their direct use often leads to inconsistent reasoning and hallucination. While hybrid LLM-symbolic planning pipelines have emerged as a more robust alternative, they typically require extensive expert intervention to refine and validate generated action schemas. It not only limits scalability but also introduces a potential for biased interpretation, as a single expert's interpretation of ambiguous natural language descriptions might not align with the user's actual intent. To address this, we propose a novel approach that constructs an action schema library to generate multiple candidates, accounting for the diverse possible interpretations of natural language descriptions. We further introduce a semantic validation and ranking module that automatically filter and rank the generated schemas and plans without expert-in-the-loop. The experiments showed our pipeline maintains superiority in planning over the direct LLM planning approach. These findings demonstrate the feasibility of a fully automated end-to-end LLM-symbolic planner that requires no expert intervention, opening up the possibility for a broader audience to engage with AI planning with less prerequisite of domain expertise.

Figures (9)

• Figure 1: An overview of direct plan generation vs. LLM-symbolic planning pipelines.
• Figure 2: Illustration of the two limitations of expert-dependent LLM-symbolic planning pipelines
• Figure 3: An overview of the proposed pipeline, it first constructs diverse action schema candidates to cover various interpretations of the natural language descriptions. Then, it filters out low-confidence candidates to ensure the generation candidates are semantically aligned with the descriptions. Lastly, it produces and ranks multiple plans using a symbolic planner. The filtering mechanism is grounded in the concept of semantic equivalence across different representations of the same content.
• Figure 4: Finetuning sentence encoder with triplet loss
• Figure 5: The pre-trained sentence encoder demonstrates semantic alignment for matched action schemas and misalignment for mismatched ones, supporting H1.