EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association

TL;DR

This paper addresses the lack of a benchmark for e-commerce script planning by formalizing EcomScript as a three-task pipeline that aligns stepwise actions with product retrieval via purchase intentions. It introduces EcomScriptBench, a large-scale dataset built from real Amazon data, linking $605{,}229$ scripts to $2.4$ million products and $24{,}010{,}870$ purchase intentions, with human-annotated gold labels for evaluation. Across extensive experiments, LLMs show substantial challenges on the three tasks, though fine-tuning and, notably, injecting purchase intentions significantly improve performance. The work provides a foundation for automated, product-enriched script planning in e-commerce and outlines practical limitations and directions for future improvements, including reproducibility, access, and multimodal enhancements.

Abstract

Goal-oriented script planning, or the ability to devise coherent sequences of actions toward specific goals, is commonly employed by humans to plan for typical activities. In e-commerce, customers increasingly seek LLM-based assistants to generate scripts and recommend products at each step, thereby facilitating convenient and efficient shopping experiences. However, this capability remains underexplored due to several challenges, including the inability of LLMs to simultaneously conduct script planning and product retrieval, difficulties in matching products caused by semantic discrepancies between planned actions and search queries, and a lack of methods and benchmark data for evaluation. In this paper, we step forward by formally defining the task of E-commerce Script Planning (EcomScript) as three sequential subtasks. We propose a novel framework that enables the scalable generation of product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. By applying our framework to real-world e-commerce data, we construct the very first large-scale EcomScript dataset, EcomScriptBench, which includes 605,229 scripts sourced from 2.4 million products. Human annotations are then conducted to provide gold labels for a sampled subset, forming an evaluation benchmark. Extensive experiments reveal that current (L)LMs face significant challenges with EcomScript tasks, even after fine-tuning, while injecting product purchase intentions improves their performance.

Figures (3)

• Figure 1: An example of a product-enriched script for planning the objective of plan an autumn-themed party with friends and family, with relevant products associated with some steps. Note that for simpler steps, such as the first two, no products are needed.
• Figure 2: An overview of our benchmark curation and evaluation pipeline for EcomScriptBench.
• Figure 3: Distribution of the number of retrieved products at each step in EcomScriptBench.