Detecting Generated Native Ads in Conversational Search

TL;DR

This paper addresses the growing risk of generated native ads in conversational search by creating the Webis Generated Native Ads 2024 dataset, which pairs queries with responses from YouChat and Copilot and GPT-4-generated ads inserted into those responses. It evaluates detection methods, finding that sentence-transformer models (MiniLM, MPNet) achieve high precision and recall (often >0.9), while LLM-based detectors lag behind, with GPT-4 performing best among them. The study demonstrates the feasibility of client-side ad-blocking for generated native ads and reveals that some advertising language exists in organic results as well, highlighting both the potential and limitations of current detection approaches. This work contributes a publicly usable dataset and a practical framework for defending against subtly inserted ads in generative conversational systems, paving the way for broader exploration of ad types and detection strategies in real-world deployments.

Abstract

Conversational search engines such as YouChat and Microsoft Copilot use large language models (LLMs) to generate responses to queries. It is only a small step to also let the same technology insert ads within the generated responses - instead of separately placing ads next to a response. Inserted ads would be reminiscent of native advertising and product placement, both of which are very effective forms of subtle and manipulative advertising. Considering the high computational costs associated with LLMs, for which providers need to develop sustainable business models, users of conversational search engines may very well be confronted with generated native ads in the near future. In this paper, we thus take a first step to investigate whether LLMs can also be used as a countermeasure, i.e., to block generated native ads. We compile the Webis Generated Native Ads 2024 dataset of queries and generated responses with automatically inserted ads, and evaluate whether LLMs or fine-tuned sentence transformers can detect the ads. In our experiments, the investigated LLMs struggle with the task but sentence transformers achieve precision and recall values above 0.9.