AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

TL;DR

AD-Bench provides a realistic benchmark for evaluating and improving advertising marketing agents, the leaderboard and code can be found at https://github.com/Emanual20/adbench-leaderboard.

Abstract

While Large Language Model (LLM) agents have achieved remarkable progress in complex reasoning tasks, evaluating their performance in real-world environments has become a critical problem. Current benchmarks, however, are largely restricted to idealized simulations, failing to address the practical demands of specialized domains like advertising and marketing analytics. In these fields, tasks are inherently more complex, often requiring multi-round interaction with professional marketing tools. To address this gap, we propose AD-Bench, a benchmark designed based on real-world business requirements of advertising and marketing platforms. AD-Bench is constructed from real user marketing analysis requests, with domain experts providing verifiable reference answers and corresponding reference tool-call trajectories. The benchmark categorizes requests into three difficulty levels (L1-L3) to evaluate agents' capabilities under multi-round, multi-tool collaboration. Experiments show that on AD-Bench, Gemini-3-Pro achieves Pass@1 = 68.0% and Pass@3 = 83.0%, but performance drops significantly on L3 to Pass@1 = 49.4% and Pass@3 = 62.1%, with a trajectory coverage of 70.1%, indicating that even state-of-the-art models still exhibit substantial capability gaps in complex advertising and marketing analysis scenarios. AD-Bench provides a realistic benchmark for evaluating and improving advertising marketing agents, the leaderboard and code can be found at https://github.com/Emanual20/adbench-leaderboard.

Figures (11)

• Figure 1: Model performance comparison on hard queries (L3). Even state-of-the-art models struggle with complex multi-step reasoning tasks, with the best model achieving only 69% Pass@3 accuracy.
• Figure 2: System overview of AD-Bench. The online advertising environment (left) yields human-validated ground-truth trajectories, while the offline evaluation environment (right) evaluates LLM agents and uses LLM-as-a-judge to assess answer correctness and trajectory coverage.
• Figure 3: Correlation between Pass@1 and Trajectory Coverage across models at different difficulty levels. Each point is one model; the line indicates a linear fit. We observe consistent positive correlations overall ($r=0.691$), especially for L1 ($r=0.784$) and L3 ($r=0.801$), supporting that trajectory coverage is a meaningful proxy of agent evaluation quality.
• Figure 4: Trajectory coverage across difficulty levels (L1, L2, L3), illustrating the distribution of execution patterns and their coverage rates.
• Figure 5: A typical tool invocation tree. Solid arrows indicate direct dependencies; dashed arrows indicate indirect dependencies (domain knowledge and benchmarks feed into computation).