Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv, Junyang Lin

TL;DR

DeepPlanning addresses the need for evaluating long-horizon, agentic planning under global constraints by introducing a two-domain benchmark (Travel Planning and Shopping Planning) with a three-stage data-generation pipeline and verifiable, offline sandboxes. It formalizes three core competencies—proactive information acquisition, local constrained reasoning, and global constrained optimization—and provides code-based evaluation to reveal fundamental limitations of frontier LLMs in sustained planning. Across diverse model families, results show planning fragility, where partial correctness does not guarantee executable plans, and demonstrate that internal reasoning enhances both effectiveness and efficiency. The work highlights key directions for strengthening agentic LLMs over extended horizons and provides a practical, open-source platform to drive advances in grounded, long-horizon planning.

Abstract

While agent evaluation has shifted toward long-horizon tasks, most benchmarks still emphasize local, step-level reasoning rather than the global constrained optimization (e.g., time and financial budgets) that demands genuine planning ability. Meanwhile, existing LLM planning benchmarks underrepresent the active information gathering and fine-grained local constraints typical of real-world settings. To address this, we introduce DeepPlanning, a challenging benchmark for practical long-horizon agent planning. It features multi-day travel planning and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization. Evaluations on DeepPlanning show that even frontier agentic LLMs struggle with these problems, highlighting the importance of reliable explicit reasoning patterns and parallel tool use for achieving better effectiveness-efficiency trade-offs. Error analysis further points to promising directions for improving agentic LLMs over long planning horizons. We open-source the code and data to support future research.

DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

TL;DR

DeepPlanning addresses the need for evaluating long-horizon, agentic planning under global constraints by introducing a two-domain benchmark (Travel Planning and Shopping Planning) with a three-stage data-generation pipeline and verifiable, offline sandboxes. It formalizes three core competencies—proactive information acquisition, local constrained reasoning, and global constrained optimization—and provides code-based evaluation to reveal fundamental limitations of frontier LLMs in sustained planning. Across diverse model families, results show planning fragility, where partial correctness does not guarantee executable plans, and demonstrate that internal reasoning enhances both effectiveness and efficiency. The work highlights key directions for strengthening agentic LLMs over extended horizons and provides a practical, open-source platform to drive advances in grounded, long-horizon planning.

Abstract

While agent evaluation has shifted toward long-horizon tasks, most benchmarks still emphasize local, step-level reasoning rather than the global constrained optimization (e.g., time and financial budgets) that demands genuine planning ability. Meanwhile, existing LLM planning benchmarks underrepresent the active information gathering and fine-grained local constraints typical of real-world settings. To address this, we introduce DeepPlanning, a challenging benchmark for practical long-horizon agent planning. It features multi-day travel planning and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization. Evaluations on DeepPlanning show that even frontier agentic LLMs struggle with these problems, highlighting the importance of reliable explicit reasoning patterns and parallel tool use for achieving better effectiveness-efficiency trade-offs. Error analysis further points to promising directions for improving agentic LLMs over long planning horizons. We open-source the code and data to support future research.
Paper Structure (39 sections, 5 figures, 5 tables)

This paper contains 39 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Evaluation of Agentic Tool Use
  4. Evaluation of LLM Planning
  5. DeepPlanning
  6. Benchmark Overview
  7. Proactive Information Acquisition
  8. Local Constrained Reasoning
  9. Global Constrained Optimization
  10. Benchmark Construction
  11. Stage 1: Database and Toolbox Design.
  12. Stage 2: Layered Task Generation.
  13. Stage 3: Manual Quality Control.
  14. Task Evaluation
  15. Travel Planning.
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Performance of frontier models on DeepPlanning, ranked by average case accuracy across Travel Planning and Shopping Planning. Dark/light bars denote reasoning versus non-reasoning models.
  • Figure 2: Overview of the proposed benchmark DeepPlanning.
  • Figure 3: Model performance versus task execution cost on Travel Planning. Performance is calculated across all tasks, while cost is measured by the average number of tool calls per task (top) and the average number of interaction turns per task (bottom).
  • Figure 4: Model performance versus task complexity in DeepPlanning. In each domain, performance is calculated across tasks at each complexity level.
  • Figure 5: Error pattern distribution of Claude-4.5-Opus (with reasoning enabled) on DeepPlanning.