ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants

TL;DR

ShopSimulator introduces a large-scale Chinese shopping sandbox that unifies evaluation and training for RL-driven LLM agents. It combines a Taobao-based catalog, task generation, and synthetic user profiles to support personalized, multi-turn shopping interactions with fine-grained product discrimination. Empirical results reveal that state-of-the-art LLMs struggle to act as reliable shopping assistants, especially in multi-turn and personalized settings, but a combination of supervised fine-tuning and reinforcement learning with a bottleneck-like strict reward significantly improves performance. The work provides practical guidance on reward design and training strategies, highlighting the challenges of long-horizon dialogue and personalization, and proposes ShopSimulator as a valuable platform for future RL-driven shopping agents.

Abstract

Large language model (LLM)-based agents are increasingly deployed in e-commerce shopping. To perform thorough, user-tailored product searches, agents should interpret personal preferences, engage in multi-turn dialogues, and ultimately retrieve and discriminate among highly similar products. However, existing research has yet to provide a unified simulation environment that consistently captures all of these aspects, and always focuses solely on evaluation benchmarks without training support. In this paper, we introduce ShopSimulator, a large-scale and challenging Chinese shopping environment. Leveraging ShopSimulator, we evaluate LLMs across diverse scenarios, finding that even the best-performing models achieve less than 40% full-success rate. Error analysis reveals that agents struggle with deep search and product selection in long trajectories, fail to balance the use of personalization cues, and to effectively engage with users. Further training exploration provides practical guidance for overcoming these weaknesses, with the combination of supervised fine-tuning (SFT) and reinforcement learning (RL) yielding significant performance improvements. Code and data will be released at https://github.com/ShopAgent-Team/ShopSimulator.

Figures (16)

• Figure 1: Illustration of ShopSimulator. On the one hand, the assistant agent needs to interpret preferences from user profile and communicate with users to understand their shopping needs; on the other hand, the agent iteratively search the database, click and review products, and ultimately to recommend suitable items.
• Figure 2: Statistics Dashboard of ShopSimulator. Fig (a) shows the 12 domains, while Fig(b) shows the number of first-level categories within each domain. Fig (c) and (d) show the train–test split of instructions for non-personalized and personalized settings. Each instruction can be used in both single-turn and multi-turn dialogues. Fig (e), (f) show the average attributes (e.g., unisex) and options (e.g., size–color combinations) per product, respectively.
• Figure 3: Error statistics of failed trajectories based on Claude‑4‑Sonnet: (a) Attribute errors for each action step of LLM under Multi‑Turn setting; (b) Categorize trajectory errors in terms of personalization under Single-Turn & Pers setting; (c) Assess errors of the LLM-simulated Shopper. The introduction of errors is shown in Table \ref{['tab:error_type_description']}.
• Figure 4: (a) shows agents’ average action steps per task in different scenarios. (b) shows detailed performance, where $R_{\text{finish}}$ indicates whether the agent finally recommends a product, regardless of accuracy.
• Figure 5: Fig (a) shows the error rate across all trajectories in Single-Turn & Pers, with successful trajectories assumed to have no errors. Fig (b) shows the average steps per trajectory in Multi-Turn.