Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools

TL;DR

This work tackles the inadequacy of LLMs in solving complex, multi-constraint planning by coupling LLM-driven interpretation with formal SMT-based solving. The proposed framework converts natural-language constraints into constrained satisfiability problems that SMT solvers can rigorously solve, and it uses unsat cores to iteratively repair infeasible queries with user-guided feedback. It demonstrates high success rates on TravelPlanner, robust zero-shot generalization to unseen constraints and domains, and effective interactive repair across varied user behaviors. The results suggest a practical path to reliable, user-friendly planning tools that leverage formal verification without sacrificing natural-language interaction.

Abstract

Large Language Models (LLMs) struggle to directly generate correct plans for complex multi-constraint planning problems, even with self-verification and self-critique. For example, a U.S. domestic travel planning benchmark TravelPlanner was proposed in Xie et al. (2024), where the best LLM OpenAI o1-preview can only find viable travel plans with a 10% success rate given all needed information. In this work, we tackle this by proposing an LLM-based planning framework that formalizes and solves complex multi-constraint planning problems as constrained satisfiability problems, which are further consumed by sound and complete satisfiability solvers. We start with TravelPlanner as the primary use case and show that our framework achieves a success rate of 93.9% and is effective with diverse paraphrased prompts. More importantly, our framework has strong zero-shot generalizability, successfully handling unseen constraints in our newly created unseen international travel dataset and generalizing well to new fundamentally different domains. Moreover, when user input queries are infeasible, our framework can identify the unsatisfiable core, provide failure reasons, and offers personalized modification suggestions. We show that our framework can modify and solve for an average of 81.6% and 91.7% unsatisfiable queries from two datasets and prove with ablations that all key components of our framework are effective and necessary. Project page: https://sites.google.com/view/llm-rwplanning.

Figures (6)

• Figure 1: An overview of the framework. The blue region represents LLM. Given a natural language query, LLM 1) generates steps to formulate it as an SMT problem, 2) generates corresponding codes that encode the problem and call the solver. If the solver is not able to find the solution, LLM receives unsatisfiable reasons from the solver, collects information, analyzes the current situation, and offers suggestions to modify the query interactively. LLM then updates the code based on suggestions and calls the solver again to find a feasible plan.
• Figure 2: Step to Code translation example.
• Figure 3: Example of how JSON-Step prompt generalizes to unseen constraints. Yellow: unseen constraint types. Green: corresponding generated steps.
• Figure 4: An overview of the framework with JSON.
• Figure 5: Performance (success rate %) of interactive plan repair over different numbers of iterations for two datasets