Auto-bidding in real-time auctions via Oracle Imitation Learning (OIL)

TL;DR

The paper tackles auto-bidding in real-time, multi-slot second-price auctions under budget and CPA constraints. It introduces Oracle Imitation Learning (OIL), where an oracle with full campaign data provides near-optimal bidding targets for training a student auto-bidding agent through imitation learning, using a nonlinear multiple-choice knapsack problem (MCKP) formulation. The approach develops two oracle policies (oracle-slot and oracle-upgrade) with efficient greedy heuristics and analyzes their theoretical properties, including slot and upgrade efficiencies. Empirical results on the AuctionNet dataset show that OIL, especially the OIL-slot variant trained online, outperforms online and offline reinforcement learning and LP baselines, achieving significant score improvements and robust budget utilization. By shifting the training focus to solving a near-optimal optimization problem and leveraging online imitation, the method delivers strong practical performance with improved sample efficiency in real-time bidding settings.

Abstract

Online advertising has become one of the most successful business models of the internet era. Impression opportunities are typically allocated through real-time auctions, where advertisers bid to secure advertisement slots. Deciding the best bid for an impression opportunity is challenging, due to the stochastic nature of user behavior and the variability of advertisement traffic over time. In this work, we propose a framework for training auto-bidding agents in multi-slot second-price auctions to maximize acquisitions (e.g., clicks, conversions) while adhering to budget and cost-per-acquisition (CPA) constraints. We exploit the insight that, after an advertisement campaign concludes, determining the optimal bids for each impression opportunity can be framed as a multiple-choice knapsack problem (MCKP) with a nonlinear objective. We propose an "oracle" algorithm that identifies a near-optimal combination of impression opportunities and advertisement slots, considering both past and future advertisement traffic data. This oracle solution serves as a training target for a student network which bids having access only to real-time information, a method we term Oracle Imitation Learning (OIL). Through numerical experiments, we demonstrate that OIL achieves superior performance compared to both online and offline reinforcement learning algorithms, offering improved sample efficiency. Notably, OIL shifts the complexity of training auto-bidding agents from crafting sophisticated learning algorithms to solving a nonlinear constrained optimization problem efficiently.

Figures (5)

• Figure 1: Overview of OIL. At every time step, the auto-biding agent and the oracle observe the conversion probabilities of the available IOs, and bid for each of them. While the agent decides its bids only based on current and past data, the oracle also uses future conversion probabilities. The agent's bids are used to advance the campaign simulation, while the oracle's bids serve as a training target.
• Figure 2: Example IOs from the AuctionNet-dense dataset. A. Conversion probabilities for a single advertiser and advertisement campaign. B. Conversion probability at one time step (red dotted line, A) as a function of their lowest price (3rd slot). IOs above the dotted line have an expected CPA lower than the target ($K=8$).
• Figure 3: RTB simulator. At each step, the agent impersonates one of the advertisers and observes its IOs, replacing the bids of the impersonated advertiser with its own bids, while the bids of the other advertisers are kept unchanged. Slots and prices are decided according to a second-price auction.
• Figure 4: Learning curves PPO and OIL, AuctionNet-dense (left) and AuctionNet-sparse (right).
• Figure 5: A. Two-slopes regression of the inverse bidding coefficient as a function of the conversion probability. B. Same as A, but for the bidding coefficients. C Same as A and B, but for the bids, obtained by multiplying the coefficients and the conversion probabilities.

Theorems & Definitions (20)

• Definition 1
• Remark
• Definition 2
• Remark
• lemma 1
• lemma 2
• Definition 3
• Remark
• Remark
• lemma 3