Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policies

TL;DR

QGA addresses auto-bidding under complex advertiser environments with suboptimal offline data by integrating Q-value regularization into a Decision Transformer backbone and introducing a dual-exploration mechanism guided by a learned Q-value critic. The method enables joint policy imitation and action-value maximization, while safely exploring beyond data through multi-RTG conditioning and action perturbations. Empirical results across offline benchmarks, simulation environments, and large-scale online A/B tests demonstrate consistent improvements in key business metrics, including up to 3.27% Ad GMV and 2.49% Ad ROI in production. The work provides a practical, robust framework for auto-bidding that generalizes across realistic advertising dynamics and demonstrates readiness for full-scale deployment.

Abstract

With the rapid development of e-commerce, auto-bidding has become a key asset in optimizing advertising performance under diverse advertiser environments. The current approaches focus on reinforcement learning (RL) and generative models. These efforts imitate offline historical behaviors by utilizing a complex structure with expensive hyperparameter tuning. The suboptimal trajectories further exacerbate the difficulty of policy learning. To address these challenges, we proposes QGA, a novel Q-value regularized Generative Auto-bidding method. In QGA, we propose to plug a Q-value regularization with double Q-learning strategy into the Decision Transformer backbone. This design enables joint optimization of policy imitation and action-value maximization, allowing the learned bidding policy to both leverage experience from the dataset and alleviate the adverse impact of the suboptimal trajectories. Furthermore, to safely explore the policy space beyond the data distribution, we propose a Q-value guided dual-exploration mechanism, in which the DT model is conditioned on multiple return-to-go targets and locally perturbed actions. This entire exploration process is dynamically guided by the aforementioned Q-value module, which provides principled evaluation for each candidate action. Experiments on public benchmarks and simulation environments demonstrate that QGA consistently achieves superior or highly competitive results compared to existing alternatives. Notably, in large-scale real-world A/B testing, QGA achieves a 3.27% increase in Ad GMV and a 2.49% improvement in Ad ROI.

Figures (5)

• Figure 1: MAPE and performance scores of different models. Lower MAPE indicates stronger, while higher MAPE indicates poorer model imitation ability.
• Figure 2: Overall structure of QGA.
• Figure 3:
• Figure 4: Hyper-parameter Study
• Figure 5: Online A/B Test