RLTP: Reinforcement Learning to Pace for Delayed Impression Modeling in Preloaded Ads

TL;DR

A Reinforcement Learning To Pace framework RLTP is proposed, which learns a pacing agent that sequentially produces selection probabilities in the whole delivery period to satisfy guaranteed impression count, penalize over-delivery and maximize traffic value.

Abstract

To increase brand awareness, many advertisers conclude contracts with advertising platforms to purchase traffic and then deliver advertisements to target audiences. In a whole delivery period, advertisers usually desire a certain impression count for the ads, and they also expect that the delivery performance is as good as possible (e.g., obtaining high click-through rate). Advertising platforms employ pacing algorithms to satisfy the demands via adjusting the selection probabilities to traffic requests in real-time. However, the delivery procedure is also affected by the strategies from publishers, which cannot be controlled by advertising platforms. Preloading is a widely used strategy for many types of ads (e.g., video ads) to make sure that the response time for displaying after a traffic request is legitimate, which results in delayed impression phenomenon. Traditional pacing algorithms cannot handle the preloading nature well because they rely on immediate feedback signals, and may fail to guarantee the demands from advertisers. In this paper, we focus on a new research problem of impression pacing for preloaded ads, and propose a Reinforcement Learning To Pace framework RLTP. It learns a pacing agent that sequentially produces selection probabilities in the whole delivery period. To jointly optimize the two objectives of impression count and delivery performance, RLTP employs tailored reward estimator to satisfy the guaranteed impression count, penalize the over-delivery and maximize the traffic value. Experiments on large-scale industrial datasets verify that RLTP outperforms baseline pacing algorithms by a large margin. We have deployed the RLTP framework online to our advertising platform, and results show that it achieves significant uplift to core metrics including delivery completion rate and click-through rate.

Figures (9)

• Figure 1: The overall delivery procedure. Due to the preloading strategy from the publisher, the ad filled by the advertising platform cannot be immediately displayed at current request. We call this phenomenon as delayed impression.
• Figure 2: (Better viewed in color) Illustration of delayed impression phenomenon under the preloading strategy controlled by publisher. At each time window $T_i$, the actual impression ads may come from the filled ads at current $T_i$ and previous windows $\left\{T_1,T_2,\ldots,T_{i-1}\right\}$.
• Figure 3: Overview of our reinforcement learning to pace framework RLTP for impression pacing on preloaded ads.
• Figure 4: For each competitor, the trend of delivery completion rate during all 288 time windows in the delivery period.
• Figure 5: The curves of cumulative reward and delivery completion rate during training (30,000 episodes).