Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commerce

TL;DR

Prompt-based adaptation of LLMs in e-commerce is hampered by prompt engineering and drift in business needs. The authors propose EaP, a framework that automatically retrieves representative in-context examples from labeled data $\mathcal{E}$ to maximize few-shot performance, balancing global coverage and input similarity, with no model updates. They also introduce EaP_lite, which replaces natural-language prompts with labeled examples to reduce latency. Experiments across four production tasks show EaP outperforms handcrafted prompts, EaP_lite speeds up inference by up to 70%, and online A/B tests report a 0.06% composite revenue gain, demonstrating scalable, real-world impact.

Abstract

Prompting LLMs offers an efficient way to guide output generation without explicit model training. In the e-commerce domain, prompting-based applications are widely used for tasks such as query understanding, recommender systems, and customer support. However, adapting LLMs to different tasks often requires extensive prompt engineering by domain experts, along with frequent updates to align with evolving business needs. Additionally, crafting fully unbiased natural language prompts remains a challenge for humans. To address these challenges, we propose a novel framework, Examples as the Prompt (EaP) which leverages labeled data to enhance prompts. Specifically, EaP automatically selects the most representative examples to maximize the few-shot capability of LLMs. It is efficient due to its unsupervised example selection and adaptive to potential data distribution shifts. We validate EaP on four real-world production use cases, demonstrating that it achieves comparable or even superior performance comparing to hand-crafted prompts designed by domain experts. Additionally, we introduce EaP_lite, which entirely replaces the natural language components of prompts with labeled examples. EaP_lite improves LLM inference speed by up to 70% without compromising performance. Latest online A/B test shows that using EaP and EaP_lite for data labeling can bring significant composite revenue gain by 0.06%.

Figures (2)

• Figure 1: Prompt examples for search query classification. The green text represents in-context examples, while the yellow text denotes the input $q$.
• Figure 2: Visualization of global and local examples selected by EaP. While global examples represents the overall data distribution, local examples help characterize the specific input.