Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation

TL;DR

This work presents NL2Flow, a fully automated, parametric data-generation and evaluation pipeline for LLM-based workflow planning that translates natural-language problems into a structured JSON intermediate representation and formal PDDL for symbolic planning. On a dataset of $2,296$ low-difficulty problems, open-source instruct-tuned LLMs demonstrated nontrivial planning capabilities, with the best model achieving $86\%$ sound plans and $69\%$ optimal plans, and the NL-to-JSON translation providing clear performance gains. The study reveals that plan quality is highly sensitive to model and prompting strategy, while highlighting the benefits of neuro-symbolic integration and scalable, programmatic evaluation to diagnose bottlenecks and error sources as planning tasks grow in complexity. NL2Flow therefore offers a reproducible framework to evaluate and guide the development of robust, natural-language-grounded planning in agentic systems, with implications for scalable workflow automation and AI planning research.

Abstract

Robust workflow composition is critical for effective agent performance, yet progress in Large Language Model (LLM) planning and reasoning is hindered by a scarcity of scalable evaluation data. This work introduces NL2Flow, a fully automated pipeline for generating and evaluating workflow planning problems. NL2Flow generates problems parametrically in a structured intermediate representation, translating them into both natural language and formal PDDL. I evaluate several open-source, instruct-tuned LLMs on a dataset of 2296 low-difficulty problems generated by NL2Flow. Results demonstrate that the best-performing model achieved 86% success in generating valid plans and 69% in generating optimal plans (for solvable problems). Regression analysis shows that the influence of problem characteristics on plan generation is contingent on both model and prompt design. Importantly, translating natural language problems into a structured JSON representation prior to symbolic planning significantly improved success rates, suggesting a benefit from neuro-symbolic integration. These findings underscore the importance of understanding error sources within LLM reasoning as systems scale to more complex tasks. As LLM reasoning scales to increasingly complex problems, understanding the shifting bottlenecks and sources of error within these systems will be crucial.

Figures (7)

• Figure 1: Steps to obtain a plan from a planning problem in natural language. Two approaches are shown: (1) Direct planning, where a Large Language Model (LLM) Planner directly generates a plan from the input. (2) Intermediate representation planning, where an LLM Translator first converts the natural language instruction into an intermediate representation, which is then used by symbolic planning algorithms to compute a plan.
• Figure 2: Comparison of LLM performance in generating sound, valid, and optimal plans using verbose and concise prompt styles
• Figure 3: Comparison of large language model performance (accuracy) in detecting 'no plan' scenarios, evaluated with verbose and concise prompt styles
• Figure 4: Performance comparison of large language models on planning tasks with feasible plans
• Figure 5: LLM-based natural language to JSON translation for planning