SUMMA: A Multimodal Large Language Model for Advertisement Summarization

TL;DR

SUMMA tackles the challenge of understanding multimodal video ads for better query-ad matching by generating advertising-focused textual summaries from video, frames, and OCR/ASR transcripts. It introduces a two-stage training pipeline—domain-specific multimodal supervised fine-tuning (SFT) followed by mixed-reward reinforcement learning (RL)—and constructs AdSum-Doubao, AdSum-Human, and AdSum-Test datasets to support this process. Through offline and online evaluations, SUMMA demonstrates that multimodal inputs plus a mixed lexical-semantic reward improve summary quality and downstream retrieval and ranking performance, yielding a 1.5% increase in advertising revenue in production. The work presents a practical paradigm for condensing multimodal ad content into interpretable text while preserving relevance to user queries, enabling scalable integration into search advertising systems.

Abstract

Understanding multimodal video ads is crucial for improving query-ad matching and relevance ranking on short video platforms, enhancing advertising effectiveness and user experience. However, the effective utilization of multimodal information with high commercial value still largely constrained by reliance on highly compressed video embeddings-has long been inadequate. To address this, we propose SUMMA (the abbreviation of Summarizing MultiModal Ads), a multimodal model that automatically processes video ads into summaries highlighting the content of highest commercial value, thus improving their comprehension and ranking in Douyin search-advertising systems. SUMMA is developed via a two-stage training strategy-multimodal supervised fine-tuning followed by reinforcement learning with a mixed reward mechanism-on domain-specific data containing video frames and ASR/OCR transcripts, generating commercially valuable and explainable summaries. We integrate SUMMA-generated summaries into our production pipeline, directly enhancing the candidate retrieval and relevance ranking stages in real search-advertising systems. Both offline and online experiments show substantial improvements over baselines, with online results indicating a statistically significant 1.5% increase in advertising revenue. Our work establishes a novel paradigm for condensing multimodal information into representative texts, effectively aligning visual ad content with user query intent in retrieval and recommendation scenarios.

Figures (8)

• Figure 1: (a) In search advertising, multimodal ad processing is key to optimal user-ad relevance. (b) Mainstream embedding-based methods rely on coarse multimodal fusion and lack interpretability. (c) Our SUMMA extracts video ad summaries, enabling text to replace original videos—easing fusion demands and boosting system performance.
• Figure 2: The overview pipeline of our SUMMA. In Stage 1, through cold-start SFT using ad summarization data, we develop a domain-specific MLLM, called SUMMA-SFT, which demonstrates fundamental capabilities in advertisement summarization. In Stage 2, building upon SUMMA-SFT, we implement reinforcement learning via GRPO with a mixed-reward mechanism, and the resultant model, SUMMA-RL, demonstrates further enhanced capacity.
• Figure 3: The annotation rules of the multimodal ad content summarization task: both our experts and the AI tool should comply with these rules while summarizing an ad video.
• Figure 4: The prompt to request Doubao to synthesize ad video summaries.
• Figure 5: The prompt to request Doubao to verify the quality of the synthesized ad video summaries.