Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards

Yuwei Cheng, Zifeng Zhao, Haifeng Xu

TL;DR

The paper addresses learning personalized ad impact under delayed, cumulative, and heterogeneous effects by formulating ad bidding as a Contextual Markov Decision Process with delayed Poisson rewards. It introduces a two-stage maximum likelihood framework with data-splitting to estimate delayed and instantaneous effects, and develops a reinforcement learning algorithm that achieves a near-optimal regret bound of $\tilde{O}(dH^2\sqrt{T})$ with high probability. The approach is validated via simulations showing $\sqrt{T}$-scale regret and superiority over baseline bidding strategies. The work offers a principled, theoretically-grounded method for personalized, long-horizon bidding, with practical implications for auto-bidding platforms and targeted advertising.

Abstract

Online advertising platforms use automated auctions to connect advertisers with potential customers, requiring effective bidding strategies to maximize profits. Accurate ad impact estimation requires considering three key factors: delayed and long-term effects, cumulative ad impacts such as reinforcement or fatigue, and customer heterogeneity. However, these effects are often not jointly addressed in previous studies. To capture these factors, we model ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards. For efficient estimation, we propose a two-stage maximum likelihood estimator combined with data-splitting strategies, ensuring controlled estimation error based on the first-stage estimator's (in)accuracy. Building on this, we design a reinforcement learning algorithm to derive efficient personalized bidding strategies. This approach achieves a near-optimal regret bound of $\tilde{O}{(dH^2\sqrt{T})}$, where $d$ is the contextual dimension, $H$ is the number of rounds, and $T$ is the number of customers. Our theoretical findings are validated by simulation experiments.

Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards

TL;DR

The paper addresses learning personalized ad impact under delayed, cumulative, and heterogeneous effects by formulating ad bidding as a Contextual Markov Decision Process with delayed Poisson rewards. It introduces a two-stage maximum likelihood framework with data-splitting to estimate delayed and instantaneous effects, and develops a reinforcement learning algorithm that achieves a near-optimal regret bound of with high probability. The approach is validated via simulations showing -scale regret and superiority over baseline bidding strategies. The work offers a principled, theoretically-grounded method for personalized, long-horizon bidding, with practical implications for auto-bidding platforms and targeted advertising.

Abstract

Online advertising platforms use automated auctions to connect advertisers with potential customers, requiring effective bidding strategies to maximize profits. Accurate ad impact estimation requires considering three key factors: delayed and long-term effects, cumulative ad impacts such as reinforcement or fatigue, and customer heterogeneity. However, these effects are often not jointly addressed in previous studies. To capture these factors, we model ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards. For efficient estimation, we propose a two-stage maximum likelihood estimator combined with data-splitting strategies, ensuring controlled estimation error based on the first-stage estimator's (in)accuracy. Building on this, we design a reinforcement learning algorithm to derive efficient personalized bidding strategies. This approach achieves a near-optimal regret bound of , where is the contextual dimension, is the number of rounds, and is the number of customers. Our theoretical findings are validated by simulation experiments.
Paper Structure (21 sections, 20 theorems, 83 equations, 2 figures, 1 table, 3 algorithms)

This paper contains 21 sections, 20 theorems, 83 equations, 2 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Modeling Personalized Ad Impact by CMDP with Delayed Rewards
  3. Learning goal: regret minimization
  4. Algorithm Design
  5. Analysis of Near-Optimal Regret Bound
  6. Experiments
  7. Discussion and Limitation
  8. Additional Discussion
  9. Related Work
  10. On the Potential to Extend the Linearity Assumption
  11. Flexibility in State Space Modeling
  12. Illustrative Example
  13. Computational Complexity
  14. Details of Omitted Algorithm
  15. Proof of Theorem \ref{['thm:main_theorem']}
  16. ...and 6 more sections

Key Result

Lemma 3.3

Given $l \in \mathcal{H}$, with probability at least $1-\delta$, $\hat{\bm{\theta}}^t_l$ defined in Eqn. eq:theta satisfies $\|\bm{\theta}_l - \hat{\bm{\theta}}^t_l\|^2_{\mathbf{V}^t_l} \leq \gamma, \forall t \geq 0$, where $\gamma = 896 d \bm{B}_{x}\bm{B}_{\theta}(1+\bm{B}_{x}\bm{B}_{\theta})\log\l

Figures (2)

  • Figure 1: Illustration
  • Figure 2: Illustration of Learner-Customer Interaction in CMDP

Theorems & Definitions (33)

  • Definition 2.1: State
  • Remark 2.3
  • Definition 2.4: Transition
  • Remark 2.5
  • Definition 3.1
  • Remark 3.2: Key Tuning Parameters for Algorithm \ref{['algo']}
  • Lemma 3.3: Theorem 1 in xue2024efficient
  • Theorem 4.1: Confidence Region for $\hat{d}^t_l$
  • Theorem 4.2: Nearly Optimal Regret
  • Lemma 4.3: Term (i) Upper Bound
  • ...and 23 more