MLLM-VADStory: Domain Knowledge-Driven Multimodal LLMs for Video Ad Storyline Insights

TL;DR

This paper introduces MLLM-VADStory, a domain knowledge-driven framework that uses multimodal large language models to quantify and interpret video ad storytelling at scale. By segmenting ads into functional units, classifying each unit's role via a novel advertising-specific taxonomy, and summarizing functional sequences, the approach uncovers data-driven storyline structures and links them to performance metrics. Applied to 50k ads across four subverticals, the framework reveals that storytelling improves video retention, with optimal story arcs varying by industry; early hook-focused and problem-driven sequences tend to boost dwell time, while social proof and clear solutions enhance conversions. The work demonstrates the value of injecting domain knowledge into MLLMs for scalable, interpretable insights into video creatives, offering practical guidance for advertisers and a foundation for future domain-guided video understanding research.

Abstract

We propose MLLM-VADStory, a novel domain knowledge-guided multimodal large language models (MLLM) framework to systematically quantify and generate insights for video ad storyline understanding at scale. The framework is centered on the core idea that ad narratives are structured by functional intent, with each scene unit performing a distinct communicative function, delivering product and brand-oriented information within seconds. MLLM-VADStory segments ads into functional units, classifies each unit's functionality using a novel advertising-specific functional role taxonomy, and then aggregates functional sequences across ads to recover data-driven storyline structures. Applying the framework to 50k social media video ads across four industry subverticals, we find that story-based creatives improve video retention, and we recommend top-performing story arcs to guide advertisers in creative design. Our framework demonstrates the value of using domain knowledge to guide MLLMs in generating scalable insights for video ad storylines, making it a versatile tool for understanding video creatives in general.

Figures (4)

• Figure 1: Video Analysis Pipeline for Story Understanding
• Figure 2: An example storyline for a video creative
• Figure 3: Illustration of Functional Sequence Summarization and Example Storyline Structures
• Figure 4: % Change in Dwell Rate in the First 10 Seconds when Ads Contain Stories