Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding

Haoming Li, Yusen Huo, Shuai Dou, Zhenzhe Zheng, Zhilin Zhang, Chuan Yu, Jian Xu, Fan Wu

TL;DR

The paper tackles the sim2real gap in auto-bidding by proposing an iterative offline RL framework that emphasizes trajectory-wise exploration and exploitation. It introduces Trajectory-wise Exploration via Parameter Space Noise (PSN) and Robust Trajectory Weighting for exploitation, along with Reward-model-based robustness to reward stochasticity, to overcome offline conservatism. Safety during online exploration is ensured by Safe Exploration by Adaptive Action Selection (SEAS), which adaptively chooses between exploratory and safe actions with a provable performance bound $(1-\epsilon)J_s$. Empirical results in simulated environments and Alibaba’s platform show improved policy performance, dataset quality, and safety, with near-expert performance achieved in few iterations. The approach has practical significance for industrial RL systems requiring rapid, safe online policy improvements from large-scale offline data.

Abstract

In online advertising, advertisers participate in ad auctions to acquire ad opportunities, often by utilizing auto-bidding tools provided by demand-side platforms (DSPs). The current auto-bidding algorithms typically employ reinforcement learning (RL). However, due to safety concerns, most RL-based auto-bidding policies are trained in simulation, leading to a performance degradation when deployed in online environments. To narrow this gap, we can deploy multiple auto-bidding agents in parallel to collect a large interaction dataset. Offline RL algorithms can then be utilized to train a new policy. The trained policy can subsequently be deployed for further data collection, resulting in an iterative training framework, which we refer to as iterative offline RL. In this work, we identify the performance bottleneck of this iterative offline RL framework, which originates from the ineffective exploration and exploitation caused by the inherent conservatism of offline RL algorithms. To overcome this bottleneck, we propose Trajectory-wise Exploration and Exploitation (TEE), which introduces a novel data collecting and data utilization method for iterative offline RL from a trajectory perspective. Furthermore, to ensure the safety of online exploration while preserving the dataset quality for TEE, we propose Safe Exploration by Adaptive Action Selection (SEAS). Both offline experiments and real-world experiments on Alibaba display advertising platform demonstrate the effectiveness of our proposed method.

Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding

TL;DR

The paper tackles the sim2real gap in auto-bidding by proposing an iterative offline RL framework that emphasizes trajectory-wise exploration and exploitation. It introduces Trajectory-wise Exploration via Parameter Space Noise (PSN) and Robust Trajectory Weighting for exploitation, along with Reward-model-based robustness to reward stochasticity, to overcome offline conservatism. Safety during online exploration is ensured by Safe Exploration by Adaptive Action Selection (SEAS), which adaptively chooses between exploratory and safe actions with a provable performance bound . Empirical results in simulated environments and Alibaba’s platform show improved policy performance, dataset quality, and safety, with near-expert performance achieved in few iterations. The approach has practical significance for industrial RL systems requiring rapid, safe online policy improvements from large-scale offline data.

Abstract

In online advertising, advertisers participate in ad auctions to acquire ad opportunities, often by utilizing auto-bidding tools provided by demand-side platforms (DSPs). The current auto-bidding algorithms typically employ reinforcement learning (RL). However, due to safety concerns, most RL-based auto-bidding policies are trained in simulation, leading to a performance degradation when deployed in online environments. To narrow this gap, we can deploy multiple auto-bidding agents in parallel to collect a large interaction dataset. Offline RL algorithms can then be utilized to train a new policy. The trained policy can subsequently be deployed for further data collection, resulting in an iterative training framework, which we refer to as iterative offline RL. In this work, we identify the performance bottleneck of this iterative offline RL framework, which originates from the ineffective exploration and exploitation caused by the inherent conservatism of offline RL algorithms. To overcome this bottleneck, we propose Trajectory-wise Exploration and Exploitation (TEE), which introduces a novel data collecting and data utilization method for iterative offline RL from a trajectory perspective. Furthermore, to ensure the safety of online exploration while preserving the dataset quality for TEE, we propose Safe Exploration by Adaptive Action Selection (SEAS). Both offline experiments and real-world experiments on Alibaba display advertising platform demonstrate the effectiveness of our proposed method.
Paper Structure (17 sections, 2 theorems, 14 equations, 6 figures, 4 tables, 1 algorithm)

This paper contains 17 sections, 2 theorems, 14 equations, 6 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Performance Bottleneck of iterative offline RL
  5. Proposed Framework
  6. Trajectory-wise Exploration
  7. Trajectory-wise Exploitation
  8. Safe Exploration
  9. Experiments
  10. Overall Performance in a Simulated Environment
  11. Online Experiments
  12. Ablation Studies
  13. Conclusion
  14. Theoretical Justification of Robust Trajectory Weighting
  15. Proof of Theorem \ref{['thm:SEAS']}
  16. ...and 2 more sections

Key Result

theorem 1

For any policy $\pi_e$ and any $\epsilon \in (0,1)$, given safe policies $\{\pi_s^i\}_{i=1}^n$ that satisfy $J(\pi_s^i)\ge J_s,\forall 1\le i\le n$, the expected return $\mathbb E_\tau[R(\tau)]$ of trajectories generated by SEAS satisfies $\mathbb E_\tau[R(\tau)] \ge (1-\epsilon)J_s$.

Figures (6)

  • Figure 1: Iterative offline RL with Trajectory-wise Exploration and Exploitation (TEE) and Safe Exploration by Adaptive Action Selection (SEAS). Components proposed in this work are highlighted in red.
  • Figure 2: Performance of exploration policies with different noise scale, as well as the performance of policies trained with IQL on datasets collected by those exploration policies.
  • Figure 3: Comparison of trajectory return distributions of datasets collected by ASN and PSN.
  • Figure 4: Overall performance in a simulated environment.
  • Figure 5: Safety constraint satisfaction.
  • ...and 1 more figures

Theorems & Definitions (2)

  • theorem 1
  • theorem 1