On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

TL;DR

The paper conducts a structured evaluation of OpenAI's o1 planning models across feasibility, optimality, and generalizability, benchmarking against GPT-4 and o1-mini on six diverse planning domains. It reveals that o1-preview excels at constraint adherence and maintaining plan-state, but struggles with optimality and generalization in spatially complex tasks. The authors argue for improved memory management, self-evaluation, and integration of external memory, symbolic verification, and multimodal data to push LLM-based planners toward robust, real-world planning. This pilot study provides actionable directions for future research to enhance efficiency, adaptability, and generalization in LLM-driven planning systems.

Abstract

Recent advancements in Large Language Models (LLMs) have showcased their ability to perform complex reasoning tasks, but their effectiveness in planning remains underexplored. In this study, we evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks, focusing on three key aspects: feasibility, optimality, and generalizability. Through empirical evaluations on constraint-heavy tasks (e.g., $\textit{Barman}$, $\textit{Tyreworld}$) and spatially complex environments (e.g., $\textit{Termes}$, $\textit{Floortile}$), we highlight o1-preview's strengths in self-evaluation and constraint-following, while also identifying bottlenecks in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions in structured environments. However, the model often generates suboptimal solutions with redundant actions and struggles to generalize effectively in spatially complex tasks. This pilot study provides foundational insights into the planning limitations of LLMs, offering key directions for future research on improving memory management, decision-making, and generalization in LLM-based planning. Code available at https://github.com/VITA-Group/o1-planning.

Figures (12)

• Figure 1: Overall comparison of GPT-4, o1-mini,and o1-preview, on key planning perspectives defined by us.
• Figure 2: Feasibility error and success rate for 6 tasks and 3 models. Overall, o1 improves the success rate for certain tasks, but many problematic issues still persist. Examples of different error types are detailedd in later figures: IR: \ref{['fig:barman fail']}, \ref{['fig:blocksworld_1']}, \ref{['fig:floortile1']}, \ref{['fig:termes']}, \ref{['fig:tyrewrold']} ; IP: \ref{['fig:floortile1']} ; MG: \ref{['fig:gripper1']}
• Figure 3: Success rate and optimality rate for Blocksworld and Grippers. Compared to GPT-4, o1 can provide more optimal plans. Example of suboptimal solutions are provided in Figures \ref{['fig:blocksworld_2']} and \ref{['fig:gripper2']}.
• Figure 4: Success rate for generalization setting. GPT-4 fails entirely on challenging generalized tasks, whereas o1 is able to solve some of them. An example of randomize domain is provided in Figure \ref{['fig:randomized_typreworld']}.
• Figure 5: A failure example for Barman. The left side contains the problem statement, while the right side shows the first lines of the solutions provided by GPT-4 and o1-mini. The GPT-4 solution fails because the rules require that one hand must be empty for "filling", while the o1-mini solution fails because the rules specify that "filling" applies only to an empty shot glass.