TravelBench: A Real-World Benchmark for Multi-Turn and Tool-Augmented Travel Planning

TL;DR

TravelBench introduces a realistic, tool-augmented benchmark for multi-turn travel planning with real-world queries and a reproducible sandbox. It includes 500 solvable multi-turn dialogues, 500 solvable single-turn queries, and 103 unsolvable requests, supported by a 100k-tool-call cache and retrieval-based cache-miss handling. An LLM-as-judge rubric, augmented by tool-call error penalties and a meta-calibration score, enables robust evaluation across diverse domains. Experimental results show that even the strongest model (GPT-5.1) achieves only 68.89 overall under penalties, underscoring the gap between current capabilities and real-world travel planning. TravelBench provides a challenging, reusable framework to advance planning, tool use, and interactive decision-making research in AI systems.

Abstract

Large language model (LLM) agents have demonstrated strong capabilities in planning and tool use. Travel planning provides a natural and high-impact testbed for these capabilities, as it requires multi-step reasoning, iterative preference elicitation through interaction, and calls to external tools under evolving constraints. Prior work has studied LLMs on travel-planning tasks, but existing settings are limited in domain coverage and multi-turn interaction. As a result, they cannot support dynamic user-agent interaction and therefore fail to comprehensively assess agent capabilities. In this paper, we introduce TravelBench, a real-world travel-planning benchmark featuring multi-turn interaction and tool use. We collect user requests from real-world scenarios and construct three subsets-multi-turn, single-turn, and unsolvable-to evaluate different aspects of agent performance. For stable and reproducible evaluation, we build a controlled sandbox environment with 10 travel-domain tools, providing deterministic tool outputs for reliable reasoning. We evaluate multiple LLMs on TravelBench and conduct an analysis of their behaviors and performance. TravelBench offers a practical and reproducible benchmark for advancing LLM agents in travel planning.