ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

TL;DR

ECom-Bench introduces a multimodal, persona-driven benchmark for evaluating LLM agents in e-commerce customer support, addressing gaps in existing frameworks that lack domain specificity, multimodal tasks, and realistic user behavior. It combines persona-based user simulations, authentic task data, and a Model Context Protocol (MCP) toolset to rigorously test agent capabilities across diverse business scenarios. The study finds that decoupled LLM-as-Tool architectures outperform end-to-end planning models, with GPT-4o achieving the strongest but still imperfect results, revealing substantial room for improvement in reliability and consistency. The benchmark enables principled analysis of tool usage, memory, and user-behavior realism, offering practical support for advancing intelligent customer service systems in real-world e-commerce settings.

Abstract

In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.

Figures (6)

• Figure 1: Framwork of ECom-Bench
• Figure 2: Distribution of task instances
• Figure 3: pass^ 8 across models
• Figure 4: pass^ 1 across models
• Figure 5: Breakdown trajectories of Doubao-1.5-Pro