Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks

Yunuo Liu, Dawei Zhu, Zena Al-Khalili, Dai Cheng, Yanjun Chen, Dietrich Klakow, Wei Zhang, Xiaoyu Shen

TL;DR

PricingLogic introduces a real-world tourism pricing benchmark with 42 pricing policies and 300 questions across three difficulty levels, designed to evaluate LLM reasoning and arithmetic under overlapping rules. The authors compare end-to-end prompting and code-assisted reasoning, finding that CaR substantially improves accuracy on both standard and bundled pricing tasks, though complex scenarios remain challenging. Key findings show state-of-the-art models struggle to reliably automate revenue-critical pricing, highlighting a gap between theoretical reasoning and practical deployment. The work provides a public dataset and code, and argues for safeguards and domain adaptation to enable trustworthy AI-assisted pricing in the travel industry.

Abstract

We present PricingLogic, the first benchmark that probes whether Large Language Models(LLMs) can reliably automate tourism-related prices when multiple, overlapping fare rules apply. Travel agencies are eager to offload this error-prone task onto AI systems; however, deploying LLMs without verified reliability could result in significant financial losses and erode customer trust. PricingLogic comprises 300 natural-language questions based on booking requests derived from 42 real-world pricing policies, spanning two levels of difficulty: (i) basic customer-type pricing and (ii)bundled-tour calculations involving interacting discounts. Evaluations of a line of LLMs reveal a steep performance drop on the harder tier,exposing systematic failures in rule interpretation and arithmetic reasoning.These results highlight that, despite their general capabilities, today's LLMs remain unreliable in revenue-critical applications without further safeguards or domain adaptation. Our code and dataset are available at https://github.com/EIT-NLP/PricingLogic.

PricingLogic: Evaluating LLMs Reasoning on Complex Tourism Pricing Tasks

TL;DR

PricingLogic introduces a real-world tourism pricing benchmark with 42 pricing policies and 300 questions across three difficulty levels, designed to evaluate LLM reasoning and arithmetic under overlapping rules. The authors compare end-to-end prompting and code-assisted reasoning, finding that CaR substantially improves accuracy on both standard and bundled pricing tasks, though complex scenarios remain challenging. Key findings show state-of-the-art models struggle to reliably automate revenue-critical pricing, highlighting a gap between theoretical reasoning and practical deployment. The work provides a public dataset and code, and argues for safeguards and domain adaptation to enable trustworthy AI-assisted pricing in the travel industry.

Abstract

We present PricingLogic, the first benchmark that probes whether Large Language Models(LLMs) can reliably automate tourism-related prices when multiple, overlapping fare rules apply. Travel agencies are eager to offload this error-prone task onto AI systems; however, deploying LLMs without verified reliability could result in significant financial losses and erode customer trust. PricingLogic comprises 300 natural-language questions based on booking requests derived from 42 real-world pricing policies, spanning two levels of difficulty: (i) basic customer-type pricing and (ii)bundled-tour calculations involving interacting discounts. Evaluations of a line of LLMs reveal a steep performance drop on the harder tier,exposing systematic failures in rule interpretation and arithmetic reasoning.These results highlight that, despite their general capabilities, today's LLMs remain unreliable in revenue-critical applications without further safeguards or domain adaptation. Our code and dataset are available at https://github.com/EIT-NLP/PricingLogic.
Paper Structure (25 sections, 7 figures, 7 tables)

This paper contains 25 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. PricingLogic Construction
  3. Collection and Organization of Tourism Products and Discount Policies
  4. Dataset and Task Setups
  5. Task 1: Standard price policies.
  6. Task 2: Bundled price policies.
  7. Methods
  8. End-to-end prompting (E2E).
  9. Code-assisted reasoning (CaR).
  10. Experiments
  11. Experimental Setups
  12. Models.
  13. Inference settings.
  14. Metrics.
  15. Results on Task 1
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Automatic quotation use case (left) and its two LLM-based realizations (right).
  • Figure 2: E2E method prompt.
  • Figure 3: CaR method step1 prompt.
  • Figure 4: CaR method step2 prompt for booking information analysis.
  • Figure 5: CaR method step2 prompt for code arguments analysis.
  • ...and 2 more figures