Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment

Akira Kasuga, Ryo Yonetani

TL;DR

CXSimulator addresses the challenge of evaluating untested web-marketing campaigns without costly online experiments by using LLM embeddings to encode event descriptions and learn transitions between user actions. The framework constructs an event-transition graph and trains Cls and Reg to predict edge existence and probabilities, enabling offline simulation of campaign effects with a new event. Experiments on GA360 BigQuery data show that LLM-embedding-based predictors outperform Node2Vec, GAE, and GPT-based baselines, and a user study with marketers reveals meaningful correlation with expert judgments. The work demonstrates the practical potential of offline marketing assessment and opens avenues for domain-specific fine-tuning and interpretability improvements.

Abstract

This paper presents the Customer Experience (CX) Simulator, a novel framework designed to assess the effects of untested web-marketing campaigns through user behavior simulations. The proposed framework leverages large language models (LLMs) to represent various events in a user's behavioral history, such as viewing an item, applying a coupon, or purchasing an item, as semantic embedding vectors. We train a model to predict transitions between events from their LLM embeddings, which can even generalize to unseen events by learning from diverse training data. In web-marketing applications, we leverage this transition prediction model to simulate how users might react differently when new campaigns or products are presented to them. This allows us to eliminate the need for costly online testing and enhance the marketers' abilities to reveal insights. Our numerical evaluation and user study, utilizing BigQuery Public Datasets from the Google Merchandise Store, demonstrate the effectiveness of our framework.

CXSimulator: A User Behavior Simulation using LLM Embeddings for Web-Marketing Campaign Assessment

TL;DR

CXSimulator addresses the challenge of evaluating untested web-marketing campaigns without costly online experiments by using LLM embeddings to encode event descriptions and learn transitions between user actions. The framework constructs an event-transition graph and trains Cls and Reg to predict edge existence and probabilities, enabling offline simulation of campaign effects with a new event. Experiments on GA360 BigQuery data show that LLM-embedding-based predictors outperform Node2Vec, GAE, and GPT-based baselines, and a user study with marketers reveals meaningful correlation with expert judgments. The work demonstrates the practical potential of offline marketing assessment and opens avenues for domain-specific fine-tuning and interpretability improvements.

Abstract

This paper presents the Customer Experience (CX) Simulator, a novel framework designed to assess the effects of untested web-marketing campaigns through user behavior simulations. The proposed framework leverages large language models (LLMs) to represent various events in a user's behavioral history, such as viewing an item, applying a coupon, or purchasing an item, as semantic embedding vectors. We train a model to predict transitions between events from their LLM embeddings, which can even generalize to unseen events by learning from diverse training data. In web-marketing applications, we leverage this transition prediction model to simulate how users might react differently when new campaigns or products are presented to them. This allows us to eliminate the need for costly online testing and enhance the marketers' abilities to reveal insights. Our numerical evaluation and user study, utilizing BigQuery Public Datasets from the Google Merchandise Store, demonstrate the effectiveness of our framework.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CXSimulator Framework
  4. User Behavior Modeling
  5. Transition Probability Prediction
  6. Simulating Campaign Effects
  7. Experiment
  8. Numerical Evaluation
  9. User Study
  10. Limitations and Possible Extensions
  11. Conclusion

Figures (2)

  • Figure 1: CXSimulator Framework: a) Constructing an event transition graph, b) Hybrid model for regressing all transition probabilities, c) Simulating user behaviors and assessing a campaign with a control and a treatment group.
  • Figure 2: CXSimulator User Interface (UI) : Left) Input your campaigns and run a simulation. Right) Execution results.