Nash Equilibrium Constrained Auto-bidding With Bi-level Reinforcement Learning
Zhiyu Mou, Miao Xu, Rongquan Bai, Zhuoran Yang, Chuan Yu, Jian Xu, Bo Zheng
TL;DR
This work reframes auto-bidding as Nash Equilibrium Constrained Bidding (NCB), aiming to maximize social welfare under the $\epsilon$-Nash Equilibrium constraint. It introduces a Bi-level Policy Gradient (BPG) framework that uses a primal-dual approach and a penalized single-level reformulation with a unified optimizer, enabling gradients that do not scale with the number of advertisers. The authors provide theoretical guarantees and demonstrate strong empirical performance in simulations and a real-world TaoBao deployment, showing improved social welfare (GMV) and constraint compliance. Overall, the method advances platform-level optimization in multi-agent auto-bidding and offers scalable, provable mechanisms for NE selection in large-scale online advertising ecosystems.
Abstract
Many online advertising platforms provide advertisers with auto-bidding services to enhance their advertising performance. However, most existing auto-bidding algorithms fail to accurately capture the auto-bidding problem formulation that the platform truly faces, let alone solve it. Actually, we argue that the platform should try to help optimize each advertiser's performance to the greatest extent -- which makes $ε$-Nash Equilibrium ($ε$-NE) a necessary solution concept -- while maximizing the social welfare of all the advertisers for the platform's long-term value. Based on this, we introduce the \emph{Nash-Equilibrium Constrained Bidding} (NCB), a new formulation of the auto-bidding problem from the platform's perspective. Specifically, it aims to maximize the social welfare of all advertisers under the $ε$-NE constraint. However, the NCB problem presents significant challenges due to its constrained bi-level structure and the typically large number of advertisers involved. To address these challenges, we propose a \emph{Bi-level Policy Gradient} (BPG) framework with theoretical guarantees. Notably, its computational complexity is independent of the number of advertisers, and the associated gradients are straightforward to compute. Extensive simulated and real-world experiments validate the effectiveness of the BPG framework.