MindMem: Multimodal for Predicting Advertisement Memorability Using LLMs and Deep Learning

TL;DR

MindMem introduces a multimodal framework that predicts advertisement memorability by integrating textual, visual, and auditory information through a sequential pipeline of projection, self-attention pooling, cross-attention, and fusion. It achieves state-of-the-art performance on the LAMBDA and Memento10K benchmarks and demonstrates the benefit of neuro-inspired attention for cross-modal integration. The authors further present MindMem-ReAd, an LLM-driven system that regenerates advertisements to boost memorability, with substantial gains for low-memorability content. Together, these contributions show the potential of combining multimodal perception with generative AI to optimize advertising strategies in multi-agent settings.

Abstract

In the competitive landscape of advertising, success hinges on effectively navigating and leveraging complex interactions among consumers, advertisers, and advertisement platforms. These multifaceted interactions compel advertisers to optimize strategies for modeling consumer behavior, enhancing brand recall, and tailoring advertisement content. To address these challenges, we present MindMem, a multimodal predictive model for advertisement memorability. By integrating textual, visual, and auditory data, MindMem achieves state-of-the-art performance, with a Spearman's correlation coefficient of 0.631 on the LAMBDA and 0.731 on the Memento10K dataset, consistently surpassing existing methods. Furthermore, our analysis identified key factors influencing advertisement memorability, such as video pacing, scene complexity, and emotional resonance. Expanding on this, we introduced MindMem-ReAd (MindMem-Driven Re-generated Advertisement), which employs Large Language Model-based simulations to optimize advertisement content and placement, resulting in up to a 74.12% improvement in advertisement memorability. Our results highlight the transformative potential of Artificial Intelligence in advertising, offering advertisers a robust tool to drive engagement, enhance competitiveness, and maximize impact in a rapidly evolving market.

Figures (14)

• Figure 1: Architecture of the MindMem model for predicting advertisement memorability. The model processes visual, auditory, and textual inputs using pre-trained embedding models (indicated by snowflake icons), such as encoder LLMs for audio, video, and text, which remain non-trainable (frozen) during training. These embeddings are then fed into trainable components (indicated by fire icons). The trainable layers include projection layers that align embeddings into a shared latent space, multi-head self-attention layers that capture intra-modal dependencies, and multi-head cross-attention layers that integrate information across modalities. Finally, a fusion layer combines the attended embeddings to predict the memorability score.
• Figure 2: Performance comparison of MindMem with four state-of-the-art methods: Henry, 10-shot GPT3.5, 10-shot GPT4.0-o, and Vit-Mem. MindMem consistently outperformed the others, achieving the highest average accuracy and Spearman’s correlation coefficient ($\rho$).
• Figure 3: Relationship between content factors and memorability scores on the LAMBDA samples in a test set. (a) video pace, (b) number of scenes, (c) number of emotions in a video, (d) video orientation, (e) video duration and (f) number of color themes are compared with predicted memorability scores. Statistical significance is measured using one-way ANOVA test (a and b), and T-test (d). Spearman’s correlation coefficient is displayed for scatter plots (c, e, and f).
• Figure 4: Advertisement generated by MindMem-ReAd (Advertisement #1). Images are created using Ideogram (https://ideogram.ai)
• Figure 5: Advertisement generated by MindMem-ReAd (Advertisement #2). Images are created using Ideogram (https://ideogram.ai)