Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce

Alberto Castelo, Zahra Zanjani Foumani, Ailin Fan, Keat Yang Koay, Vibhor Malik, Yuanzheng Zhu, Han Li, Meysam Feghhi, Ronie Uliana, Shuang Xie, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Francis Pelland, Jonathan Faerman, Nikolas LeBlanc, Aaron Glazer, Andrew McNamara, Lingyun Wang, Zhong Wu

TL;DR

SimGym is introduced, a scalable system for rapid offline A/B testing using traffic-grounded synthetic buyers powered by Large Language Model agents operating in a live browser and validated against real human outcomes from real UI changes on a major e-commerce platform under confounder control.

Abstract

A/B testing remains the gold standard for evaluating e-commerce UI changes, yet it diverts traffic, takes weeks to reach significance, and risks harming user experience. We introduce SimGym, a scalable system for rapid offline A/B testing using traffic-grounded synthetic buyers powered by Large Language Model agents operating in a live browser. SimGym extracts per-shop buyer profiles and intents from production interaction data, identifies distinct behavioral archetypes, and simulates cohort-weighted sessions across control and treatment storefronts. We validate SimGym against real human outcomes from real UI changes on a major e-commerce platform under confounder control. Even without alignment post training, SimGym agents achieve state of the art alignment with observed outcome shifts and reduces experiment cycles from weeks to under an hour , enabling rapid experimentation without exposure to real buyers.

SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce

TL;DR

SimGym is introduced, a scalable system for rapid offline A/B testing using traffic-grounded synthetic buyers powered by Large Language Model agents operating in a live browser and validated against real human outcomes from real UI changes on a major e-commerce platform under confounder control.

Abstract

A/B testing remains the gold standard for evaluating e-commerce UI changes, yet it diverts traffic, takes weeks to reach significance, and risks harming user experience. We introduce SimGym, a scalable system for rapid offline A/B testing using traffic-grounded synthetic buyers powered by Large Language Model agents operating in a live browser. SimGym extracts per-shop buyer profiles and intents from production interaction data, identifies distinct behavioral archetypes, and simulates cohort-weighted sessions across control and treatment storefronts. We validate SimGym against real human outcomes from real UI changes on a major e-commerce platform under confounder control. Even without alignment post training, SimGym agents achieve state of the art alignment with observed outcome shifts and reduces experiment cycles from weeks to under an hour , enabling rapid experimentation without exposure to real buyers.
Paper Structure (19 sections, 13 figures, 7 tables)

This paper contains 19 sections, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Simgym Platform
  3. Intent and Persona Generation Pipeline
  4. Agent Architecture
  5. Ground Truth Generation and Evaluation
  6. Ground Truth
  7. Evaluation
  8. Experiments and Results
  9. Agent Sample Size Selection
  10. Ablation Studies
  11. Memory Ablation
  12. Persona Ablation
  13. Conclusion and Future Work
  14. Simgym Platform
  15. Intent and Persona Generation
  16. ...and 4 more sections

Figures (13)

  • Figure 1: SimGym Framework Overview. Production clickstream data feeds into the persona generation pipeline (\ref{['sec: intent_persona_pipeline']}), which outputs synthetic agent personas. These agents browse both control and treatment storefronts in a live browser (\ref{['sec: agent_arch']}). Evaluation compares agent A2C rates against human outcomes via alignment rate and Pearson correlation (\ref{['sec: eval']}).
  • Figure 2: Dataset Distribution Across $20$ Storefronts. Left: Geographic distribution by country. Right: Product category mix across retail domains.
  • Figure 3: Effects of Agent Sample Size on Evaluation Metrics.
  • Figure 4: Predictive Validity Across Configurations.
  • Figure 5: Visualization of the Persona Generation Pipeline.
  • ...and 8 more figures