LLM-based Fusion of Multi-modal Features for Commercial Memorability Prediction

TL;DR

The paper tackles predicting commercial memorability by integrating multi-modal signals from financial ads using a two-branch approach: a gradient-boosted baseline and a Gemma-3 based multimodal fusion (Gemma Fusion) that projects external visual and text features into an LLM with LoRA fine-tuning. A novel aspect is the use of LLM-generated rationales as prompts to guide the fusion process, which improves robustness and generalization on the final test set, particularly for brand memorability. Key findings indicate that the LLM-based fusion can outperform the baseline under small-sample conditions, with rationales boosting brand memorability, while subtitle-derived summaries support memorability scoring. The work highlights the potential of prompting strategies and data-efficient multimodal fusion for memorability tasks and suggests avenues for gathering more data and domain-specific prompts to further enhance performance.

Abstract

This paper addresses the prediction of commercial (brand) memorability as part of "Subtask 2: Commercial/Ad Memorability" within the "Memorability: Predicting movie and commercial memorability" task at the MediaEval 2025 workshop competition. We propose a multimodal fusion system with a Gemma-3 LLM backbone that integrates pre-computed visual (ViT) and textual (E5) features by multi-modal projections. The model is adapted using Low-Rank Adaptation (LoRA). A heavily-tuned ensemble of gradient boosted trees serves as a baseline. A key contribution is the use of LLM-generated rationale prompts, grounded in expert-derived aspects of memorability, to guide the fusion model. The results demonstrate that the LLM-based system exhibits greater robustness and generalization performance on the final test set, compared to the baseline. The paper's codebase can be found at https://github.com/dsgt-arc/mediaeval-2025-memorability