Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?

Huaixiao Tou, Ying Zeng, Cong Ma, Muzhi Li, Minghao Li, Weijie Yuan, He Zhang, Kai Jia

TL;DR

ShoppingComp introduces a real-world, rubric-driven benchmark for evaluating LLM-powered shopping agents across three intertwined tasks: browsing real products, generating expert-aligned reports, and safety-critical decision making. It combines live web data with rubric-based scoring and LLM-verifier assessments to ensure verifiability and safety, covering 1,026 scenarios curated by 35 experts. Empirical results show meaningful gaps between current models (e.g., GPT-5) and human performance, particularly in product retrieval and safety reasoning, despite strong report generation and RV. The work highlights the need for robust retrieval pipelines, explicit safety checks, and open-world evaluation to advance deployable, trustworthy e-commerce agents.

Abstract

We present ShoppingComp, a challenging real-world benchmark for rigorously evaluating LLM-powered shopping agents on three core capabilities: precise product retrieval, expert-level report generation, and safety critical decision making. Unlike prior e-commerce benchmarks, ShoppingComp introduces highly complex tasks under the principle of guaranteeing real products and ensuring easy verifiability, adding a novel evaluation dimension for identifying product safety hazards alongside recommendation accuracy and report quality. The benchmark comprises 120 tasks and 1,026 scenarios, curated by 35 experts to reflect authentic shopping needs. Results reveal stark limitations of current LLMs: even state-of-the-art models achieve low performance (e.g., 11.22% for GPT-5, 3.92% for Gemini-2.5-Flash). These findings highlight a substantial gap between research benchmarks and real-world deployment, where LLMs make critical errors such as failure to identify unsafe product usage or falling for promotional misinformation, leading to harmful recommendations. ShoppingComp fills the gap and thus establishes a new standard for advancing reliable and practical agents in e-commerce.

ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?

TL;DR

ShoppingComp introduces a real-world, rubric-driven benchmark for evaluating LLM-powered shopping agents across three intertwined tasks: browsing real products, generating expert-aligned reports, and safety-critical decision making. It combines live web data with rubric-based scoring and LLM-verifier assessments to ensure verifiability and safety, covering 1,026 scenarios curated by 35 experts. Empirical results show meaningful gaps between current models (e.g., GPT-5) and human performance, particularly in product retrieval and safety reasoning, despite strong report generation and RV. The work highlights the need for robust retrieval pipelines, explicit safety checks, and open-world evaluation to advance deployable, trustworthy e-commerce agents.

Abstract

We present ShoppingComp, a challenging real-world benchmark for rigorously evaluating LLM-powered shopping agents on three core capabilities: precise product retrieval, expert-level report generation, and safety critical decision making. Unlike prior e-commerce benchmarks, ShoppingComp introduces highly complex tasks under the principle of guaranteeing real products and ensuring easy verifiability, adding a novel evaluation dimension for identifying product safety hazards alongside recommendation accuracy and report quality. The benchmark comprises 120 tasks and 1,026 scenarios, curated by 35 experts to reflect authentic shopping needs. Results reveal stark limitations of current LLMs: even state-of-the-art models achieve low performance (e.g., 11.22% for GPT-5, 3.92% for Gemini-2.5-Flash). These findings highlight a substantial gap between research benchmarks and real-world deployment, where LLMs make critical errors such as failure to identify unsafe product usage or falling for promotional misinformation, leading to harmful recommendations. ShoppingComp fills the gap and thus establishes a new standard for advancing reliable and practical agents in e-commerce.

Paper Structure

This paper contains 38 sections, 3 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data collection and verification
  4. Task Definition
  5. Data Collection
  6. LLM-as-a-Judge Verification
  7. Human Performance And Dataset Diversity
  8. Evaluations
  9. Grading
  10. Browse Products Score
  11. Report Score
  12. Safety Rubric Pass Rate
  13. Evaluation of models
  14. Models and Settings
  15. Performance
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Leaderboard comparison on the four evaluation dimensions of ShoppingComp. Top-left: Product retrieval (AnswerMatch-F1). Top-right: Scenario Coverage-F1 for report comprehensiveness. Bottom-left: Report Rationale Validity. Bottom-right: Safety Rubric Pass Rate.
  • Figure 2: Examples from ShoppingComp, including user-authored, expert-authored, and safety-critical questions. Each instance links to verified products and rubrics with supporting evidence, ensuring realism and explicit safety evaluation.
  • Figure 3: Human-in-the-loop workflow for constructing the ShoppingComp benchmark.
  • Figure 4: Distribution of Time Spent on Questions by Experts and Annotators.
  • Figure 5: Distribution of categories of ShoppingComp.