Simulation-Based Benchmarking of Reinforcement Learning Agents for Personalized Retail Promotions

TL;DR

This work tackles benchmarking reinforcement learning agents for personalized coupon targeting in retail, addressing the challenge of sparse purchase events that hinder learning. It advances by integrating a RetailSynth-based simulation with offline training and evaluation pipelines to compare contextual bandits and deep RL against static baselines. Key findings show that contextual bandits and PPO outperform static policies, with PPO achieving the best revenue and lower discounting, while NB and DQN struggle under sparse rewards; the results also indicate that simpler models can be competitive with modest data. By offering a reusable, open benchmarking blueprint for simulation-based retail AI, the work invites broader development of simulation tools and guided deployment decisions, pointing to future directions like richer customer modeling and scalable action spaces.

Abstract

The development of open benchmarking platforms could greatly accelerate the adoption of AI agents in retail. This paper presents comprehensive simulations of customer shopping behaviors for the purpose of benchmarking reinforcement learning (RL) agents that optimize coupon targeting. The difficulty of this learning problem is largely driven by the sparsity of customer purchase events. We trained agents using offline batch data comprising summarized customer purchase histories to help mitigate this effect. Our experiments revealed that contextual bandit and deep RL methods that are less prone to over-fitting the sparse reward distributions significantly outperform static policies. This study offers a practical framework for simulating AI agents that optimize the entire retail customer journey. It aims to inspire the further development of simulation tools for retail AI systems.

Figures (6)

• Figure 1: Data flow in RetailSynth environment for evaluating coupon-targeting agents.
• Figure 2: Accumulated revenue and customer retention when applying fixed coupon policies to 100 separate simulations of 100 customers over 70 time steps. Metrics collected from last 20 time steps to match evaluation period used for agent training.
• Figure 3: (a) The average accumulated revenue of customer segments with different price sensitivities. Average price coefficient given by $\sum_i \beta_{ui}^w / dim(I)$. Customers in the price-sensitive and price-insensitive segments have average price coefficients greater than or less than the median. Diamond indicators show the mean values of the price coefficient and revenue for each segment. (b) Coupon offer probability distributions by segment shown in the bar chart. The table shows average accumulated revenue of each agent normalized relative to the random policy and the actual average coupon discount value.
• Figure 4: Mutual Information scores of manually prepared features ranked in descending order.
• Figure 5: Impact of training data size on agents' accumulated revenue normalized by the random agent.