Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

Jiangyuan Wang, Kejun Xiao, Qi Sun, Huaipeng Zhao, Tao Luo, Jian Dong Zhang, Xiaoyi Zeng

TL;DR

ShoppingBench introduces a large-scale, end-to-end e-commerce benchmark centered on grounded intents, featuring a 2.5-million-product sandbox and 3,310 instructions across four challenging tasks. It formalizes a POMDP-like evaluation framework, proposes new automatic intent-constrained metrics, and demonstrates a trajectory distillation pipeline that transfers GPT-4.1 capabilities into a smaller model via SFT and RL. Experimental results reveal substantial room for improvement, with GPT-4.1 not reaching 50% ASR and a fine-tuned Qwen3-4B achieving competitive performance after distillation. The work highlights the importance of tool usage, web knowledge, and problem decomposition for real-world shopping agents and provides a concrete path toward scalable, grounded agent evaluation in e-commerce.

Abstract

Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1.

ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents

TL;DR

ShoppingBench introduces a large-scale, end-to-end e-commerce benchmark centered on grounded intents, featuring a 2.5-million-product sandbox and 3,310 instructions across four challenging tasks. It formalizes a POMDP-like evaluation framework, proposes new automatic intent-constrained metrics, and demonstrates a trajectory distillation pipeline that transfers GPT-4.1 capabilities into a smaller model via SFT and RL. Experimental results reveal substantial room for improvement, with GPT-4.1 not reaching 50% ASR and a fine-tuned Qwen3-4B achieving competitive performance after distillation. The work highlights the importance of tool usage, web knowledge, and problem decomposition for real-world shopping agents and provides a concrete path toward scalable, grounded agent evaluation in e-commerce.

Abstract

Existing benchmarks in e-commerce primarily focus on basic user intents, such as finding or purchasing products. However, real-world users often pursue more complex goals, such as applying vouchers, managing budgets, and finding multi-products seller. To bridge this gap, we propose ShoppingBench, a novel end-to-end shopping benchmark designed to encompass increasingly challenging levels of grounded intent. Specifically, we propose a scalable framework to simulate user instructions based on various intents derived from sampled real-world products. To facilitate consistent and reliable evaluations, we provide a large-scale shopping sandbox that serves as an interactive simulated environment, incorporating over 2.5 million real-world products. Experimental results demonstrate that even state-of-the-art language agents (such as GPT-4.1) achieve absolute success rates under 50% on our benchmark tasks, highlighting the significant challenges posed by our ShoppingBench. In addition, we propose a trajectory distillation strategy and leverage supervised fine-tuning, along with reinforcement learning on synthetic trajectories, to distill the capabilities of a large language agent into a smaller one. As a result, our trained agent achieves competitive performance compared to GPT-4.1.

Paper Structure

This paper contains 58 sections, 12 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agent Benchmarks
  4. E-commerce Datasets
  5. Problem Formulation
  6. ShoppingBench Construction
  7. Grounded Shopping Intention
  8. Products Finder.
  9. Knowledge.
  10. Multi-products seller.
  11. Coupon & Budget .
  12. Shopping Sandbox Implements
  13. Search engine.
  14. Web Search engine.
  15. Intent-Grounded Instruction Generation.
  16. ...and 43 more sections

Figures (9)

  • Figure 1: An illustration to depict a real-world user instruction with complex intent. Unlike previous agent benchmarks that solely focus on basic product purchases, this example incorporates coupon usage and optimal product combination within a budget.
  • Figure 2: Construction of our ShoppingBench. Our benchmark encompasses four types of real-world user purchase intents: Products Finder, Knowledge, Multi-products seller, and Coupon & Budget , with complexity increasing progressively.
  • Figure 3: ShoppingBench provides six API tools designed to facilitate agent interaction with our shopping sandbox environment.
  • Figure 4: Breakdown of failed GPT-4.1 agent trajectories in ShoppingBench, categorized as attribute mismatch (#AM), metric issue (#MI), product missing (#PM), constraint not satisfied (#CNS), and knowledge error (#KE).
  • Figure 5: Correlation analysis between various factors and absolute success rates across different intents. Detailed can be seen in Appendix F
  • ...and 4 more figures