Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The 2025 Planning Performance of Frontier Large Language Models

Augusto B. Corrêa, André G. Pereira, Jendrik Seipp

TL;DR

The paper evaluates end-to-end planning performance of frontier LLMs (GPT-5, Gemini 2.5 Pro, DeepSeek R1) against a strong planner (LAMA) on standard and obfuscated PDDL tasks from the IPC 2023 Learning Track. It uses single-shot prompts with PDDL descriptions, validates plans with VAL, and introduces obfuscated domains and a fresh 360-task set to assess pure reasoning and avoid data contamination. GPT-5 achieves competitive coverage with LAMA on standard domains, while Gemini 2.5 Pro shows robustness to obfuscation and DeepSeek R1 experiences greater degradation, highlighting progress and remaining reliance on semantic cues. The results also reveal that frontier LLMs can produce very long, valid plans but at substantial computational and energetic costs compared to traditional planners, framing a practical trade-off for real-world deployment.

Abstract

The capacity of Large Language Models (LLMs) for reasoning remains an active area of research, with the capabilities of frontier models continually advancing. We provide an updated evaluation of the end-to-end planning performance of three frontier LLMs as of 2025, where models are prompted to generate a plan from PDDL domain and task descriptions. We evaluate DeepSeek R1, Gemini 2.5 Pro, GPT-5 and as reference the planner LAMA on a subset of domains from the most recent Learning Track of the International Planning Competition. Our results show that on standard PDDL domains, the performance of GPT-5 in terms of solved tasks is competitive with LAMA. When the PDDL domains and tasks are obfuscated to test for pure reasoning, the performance of all LLMs degrades, though less severely than previously reported for other models. These results show substantial improvements over prior generations of LLMs, reducing the performance gap to planners on a challenging benchmark.

The 2025 Planning Performance of Frontier Large Language Models

TL;DR

The paper evaluates end-to-end planning performance of frontier LLMs (GPT-5, Gemini 2.5 Pro, DeepSeek R1) against a strong planner (LAMA) on standard and obfuscated PDDL tasks from the IPC 2023 Learning Track. It uses single-shot prompts with PDDL descriptions, validates plans with VAL, and introduces obfuscated domains and a fresh 360-task set to assess pure reasoning and avoid data contamination. GPT-5 achieves competitive coverage with LAMA on standard domains, while Gemini 2.5 Pro shows robustness to obfuscation and DeepSeek R1 experiences greater degradation, highlighting progress and remaining reliance on semantic cues. The results also reveal that frontier LLMs can produce very long, valid plans but at substantial computational and energetic costs compared to traditional planners, framing a practical trade-off for real-world deployment.

Abstract

The capacity of Large Language Models (LLMs) for reasoning remains an active area of research, with the capabilities of frontier models continually advancing. We provide an updated evaluation of the end-to-end planning performance of three frontier LLMs as of 2025, where models are prompted to generate a plan from PDDL domain and task descriptions. We evaluate DeepSeek R1, Gemini 2.5 Pro, GPT-5 and as reference the planner LAMA on a subset of domains from the most recent Learning Track of the International Planning Competition. Our results show that on standard PDDL domains, the performance of GPT-5 in terms of solved tasks is competitive with LAMA. When the PDDL domains and tasks are obfuscated to test for pure reasoning, the performance of all LLMs degrades, though less severely than previously reported for other models. These results show substantial improvements over prior generations of LLMs, reducing the performance gap to planners on a challenging benchmark.

Paper Structure

This paper contains 4 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experiments
  4. Discussion

Figures (2)

  • Figure 1: End-to-end planning performance of frontier LLMs and a planner (LAMA) on standard and obfuscated planning tasks from the IPC 2023 Learning Track. For each task, we prompt a model with a single call, providing the problem description in a formal language alongside two examples. We test the models on two versions of the benchmarks: the standard problems and an "obfuscated" version where all symbolic names are replaced with random strings valmeekam-et-al-neurips2023valmeekam-et-al-neurips2023datasets. To reduce the risk of data contamination, we generated 360 fresh new tasks using the parameter distributions from the IPC 2023 test set. The generated plans are then validated using a sound tool. While GPT-5 matches the performance of LAMA in standard domains, its performance decreases when the tasks are obfuscated. Remarkably, Gemini 2.5 Pro is the only LLM that is not severely impacted by obfuscation.
  • Figure 2: A comparison of plan lengths and reasoning effort. (a) The distribution of plan lengths for tasks solved by each LLM, sorted independently. (b) The reasoning tokens used by Gemini 2.5 Pro to solve tasks in the standard vs. obfuscated settings.