Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation

TL;DR

The paper tackles the challenge of evaluating automatic ad text generation (ATG) by standardizing the ATG task and releasing CAMERA, a multimodal benchmark derived from real Japanese search ads. It formalizes the problem around inputs $x$ and $a$ with outputs $y$, optimizing the objective $p(y|a,x)$ while enforcing factual consistency and relevance to latent user needs. Through extensive experiments with nine baselines including BART, T5, and LLMs, it reveals that fine-tuning yields strong intrinsic scores whereas few-shot LLMs show promise for extrinsic, human-centric attractiveness; multimodal information both helps and complicates integration. A meta-evaluation demonstrates that existing automatic metrics align variably with human judgments—better for faithfulness and fluency than attractiveness—highlighting the need for robust evaluation and reproducible benchmarks. Collectively, CAMERA provides a path toward industry-aware, open ATG research and underscores the ongoing need to balance offline metrics with online effectiveness.

Abstract

In response to the limitations of manual ad creation, significant research has been conducted in the field of automatic ad text generation (ATG). However, the lack of comprehensive benchmarks and well-defined problem sets has made comparing different methods challenging. To tackle these challenges, we standardize the task of ATG and propose a first benchmark dataset, CAMERA, carefully designed and enabling the utilization of multi-modal information and facilitating industry-wise evaluations. Our extensive experiments with a variety of nine baselines, from classical methods to state-of-the-art models including large language models (LLMs), show the current state and the remaining challenges. We also explore how existing metrics in ATG and an LLM-based evaluator align with human evaluations.

Figures (13)

• Figure 1: Examples of our dataset, translated into English for visibility. The highlighted areas indicate the aspects of advertising appeals: , , and
• Figure 2: Percentages of novel entities included in our dataset when input information is increased.
• Figure 3: Industry-wise evaluation results.
• Figure 4: Human ranking in terms of faithfulness and fluency, respectively.
• Figure 7: Human preference evaluation for each system output, comparing to a human-created reference.