Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TripScore: Benchmarking and rewarding real-world travel planning with fine-grained evaluation

Yincen Qu, Huan Xiao, Feng Li, Gregory Li, Hui Zhou, Xiangying Dai, Xiaoru Dai

TL;DR

TripScore tackles the challenge of evaluating real-world travel planning by unifying diverse criteria into a single reward, enabling direct model comparison and reinforcement learning optimization via $\mathcal{R}(\bm{S}; \bm{\theta})$. The framework introduces four constraint families (Format, Commonsense, Soft, Personal Preference), a large-scale dataset of 4,870 queries including 219 real-world requests, and evaluation validated against expert judgments. The reward blends hard and soft components with learnable weights, and the authors demonstrate RL gains (GRPO) in itinerary feasibility and quality across multiple LLMs, supported by expert-aligned correlations (e.g., cross-validated accuracy $0.6075$, model-latent agreement $\approx 0.695$ relative to $r \approx 0.839$ human ceiling). The dataset and evaluation approach offer a scalable resource for developing reliable, engaging travel-planning systems and highlight the practical impact of fine-tuning and RL for real-world planning tasks.

Abstract

Travel planning is a valuable yet complex task that poses significant challenges even for advanced large language models (LLMs). While recent benchmarks have advanced in evaluating LLMs' planning capabilities, they often fall short in evaluating feasibility, reliability, and engagement of travel plans. We introduce a comprehensive benchmark for travel planning that unifies fine-grained criteria into a single reward, enabling direct comparison of plan quality and seamless integration with reinforcement learning (RL). Our evaluator achieves moderate agreement with travel-expert annotations (60.75%) and outperforms multiple LLM-as-judge baselines. We further release a large-scale dataset of 4,870 queries including 219 real-world, free-form requests for generalization to authentic user intent. Using this benchmark, we conduct extensive experiments across diverse methods and LLMs, including test-time computation, neuro-symbolic approaches, supervised fine-tuning, and RL via GRPO. Across base models, RL generally improves itinerary feasibility over prompt-only and supervised baselines, yielding higher unified reward scores.

TripScore: Benchmarking and rewarding real-world travel planning with fine-grained evaluation

TL;DR

TripScore tackles the challenge of evaluating real-world travel planning by unifying diverse criteria into a single reward, enabling direct model comparison and reinforcement learning optimization via . The framework introduces four constraint families (Format, Commonsense, Soft, Personal Preference), a large-scale dataset of 4,870 queries including 219 real-world requests, and evaluation validated against expert judgments. The reward blends hard and soft components with learnable weights, and the authors demonstrate RL gains (GRPO) in itinerary feasibility and quality across multiple LLMs, supported by expert-aligned correlations (e.g., cross-validated accuracy , model-latent agreement relative to human ceiling). The dataset and evaluation approach offer a scalable resource for developing reliable, engaging travel-planning systems and highlight the practical impact of fine-tuning and RL for real-world planning tasks.

Abstract

Travel planning is a valuable yet complex task that poses significant challenges even for advanced large language models (LLMs). While recent benchmarks have advanced in evaluating LLMs' planning capabilities, they often fall short in evaluating feasibility, reliability, and engagement of travel plans. We introduce a comprehensive benchmark for travel planning that unifies fine-grained criteria into a single reward, enabling direct comparison of plan quality and seamless integration with reinforcement learning (RL). Our evaluator achieves moderate agreement with travel-expert annotations (60.75%) and outperforms multiple LLM-as-judge baselines. We further release a large-scale dataset of 4,870 queries including 219 real-world, free-form requests for generalization to authentic user intent. Using this benchmark, we conduct extensive experiments across diverse methods and LLMs, including test-time computation, neuro-symbolic approaches, supervised fine-tuning, and RL via GRPO. Across base models, RL generally improves itinerary feasibility over prompt-only and supervised baselines, yielding higher unified reward scores.

Paper Structure

This paper contains 36 sections, 21 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark
  4. Overview
  5. Constraint Introduction
  6. Dataset Construction
  7. Evaluation
  8. Evaluation Validation
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Distribution analysis of error types.
  13. Performance on various trip durations.
  14. Case Study.
  15. Conclusion
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Overview of TripScore. Given a query and the relevant recalled information, planning model is expected to deliver a plan that satisfies all specified constraints. The resulting plan is then evaluated to produce a comprehensive final reward score, reflecting its overall quality and adherence to user requirements.
  • Figure 2: Format and commonsense constraints error distribution in the real-world test set.
  • Figure 3: Performance distribution across varying trip durations in the synthetic test set.
  • Figure 4: Case studies of the comparison of the itinerary quality.
  • Figure 5: Demonstration of the Grouped Relative Policy Optimization (GRPO). GRPO estimates the baseline from group scores, and normalize them into a standard value. The planning model undergoes iterative optimization by maximizing the objective function $\mathcal{L}(\pi_\theta)$.