EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce

TL;DR

This paper addresses the need to evaluate LLM-based e-commerce agents beyond final-task success by introducing EComStage, a stage-wise benchmark spanning Perception, Planning, and Action. It creates seven tasks across customer and merchant orientations with real-world data, professional annotation, and a rigorous filtering pipeline. The authors benchmark over 30 agent-capable LLMs from 1B to 235B parameters, revealing stage- and orientation-specific strengths/weaknesses and that no model excels universally. The benchmark provides actionable insights for designing robust, real-world e-commerce agents and highlights the value of intermediate reasoning evaluation.

Abstract

Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications to assist customer services in tasks such as product inquiries, recommendations, and order management. Existing benchmarks primarily evaluate whether these agents successfully complete the final task, overlooking the intermediate reasoning stages that are crucial for effective decision-making. To address this gap, we propose EComStage, a unified benchmark for evaluating agent-capable LLMs across the comprehensive stage-wise reasoning process: Perception (understanding user intent), Planning (formulating an action plan), and Action (executing the decision). EComStage evaluates LLMs through seven separate representative tasks spanning diverse e-commerce scenarios, with all samples human-annotated and quality-checked. Unlike prior benchmarks that focus only on customer-oriented interactions, EComStage also evaluates merchant-oriented scenarios, including promotion management, content review, and operational support relevant to real-world applications. We evaluate a wide range of over 30 LLMs, spanning from 1B to over 200B parameters, including open-source models and closed-source APIs, revealing stage/orientation-specific strengths and weaknesses. Our results provide fine-grained, actionable insights for designing and optimizing LLM-based agents in real-world e-commerce settings.

Figures (15)

• Figure 1: Stage-wise reasoning process of an LLM-based agent when handling an e-commerce requirement.
• Figure 2: Pipeline for dataset construction.
• Figure 3: Stage-wise and orientation-wise comparison of LLM-based agents.
• Figure 4: Guidelines for human annotators.
• Figure 5: Task-specific filtering prompt for removing sentitive information.