Scaling Up LLM Reviews for Google Ads Content Moderation

TL;DR

The paper tackles the scalability challenge of applying LLM-based content moderation to Google Ads under tight compute and labeling resources. It presents a funneling pipeline that selects likely policy-violating candidates using content and actor similarity, deduplicates and diversifies the pool, and then applies a prompt-engineered LLM with soft-prompt tuning for labeling, followed by a label propagation step to extend decisions to similar past images. A feedback loop refines the candidate selection over time, improving coverage while reducing the reviewed volume. Empirically, the approach processes 400 million ads with less than 0.1% routed to the LLM, doubles the number of positive labels via propagation, and achieves higher precision than a non-LLM baseline, removing over 15% of policy-violating impressions; cross-modal embeddings outperform uni-modal ones in clustering and propagation, and the method scales to additional policies and modalities such as video and text.

Abstract

Large language models (LLMs) are powerful tools for content moderation, but their inference costs and latency make them prohibitive for casual use on large datasets, such as the Google Ads repository. This study proposes a method for scaling up LLM reviews for content moderation in Google Ads. First, we use heuristics to select candidates via filtering and duplicate removal, and create clusters of ads for which we select one representative ad per cluster. We then use LLMs to review only the representative ads. Finally, we propagate the LLM decisions for the representative ads back to their clusters. This method reduces the number of reviews by more than 3 orders of magnitude while achieving a 2x recall compared to a baseline non-LLM model. The success of this approach is a strong function of the representations used in clustering and label propagation; we found that cross-modal similarity representations yield better results than uni-modal representations.

Figures (1)

• Figure 1: A diagram of our end-to-end solution for scaling up LLMs for content moderation.