Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proactive Detection and Calibration of Seasonal Advertisements with Multimodal Large Language Models

Hamid Eghbalzadeh, Shuai Shao, Saurabh Verma, Venugopal Mani, Hongnan Wang, Jigar Madia, Vitali Karpinchyk, Andrey Malevich

TL;DR

Based on the findings, MLMs are envisioned as a teacher for knowledge distillation, a machine labeler, and a part of the ensembled and tiered seasonality detection system, which can empower ads ranking systems with enriched seasonal information.

Abstract

A myriad of factors affect large scale ads delivery systems and influence both user experience and revenue. One such factor is proactive detection and calibration of seasonal advertisements to help with increasing conversion and user satisfaction. In this paper, we present Proactive Detection and Calibration of Seasonal Advertisements (PDCaSA), a research problem that is of interest for the ads ranking and recommendation community, both in the industrial setting as well as in research. Our paper provides detailed guidelines from various angles of this problem tested in, and motivated by a large-scale industrial ads ranking system. We share our findings including the clear statement of the problem and its motivation rooted in real-world systems, evaluation metrics, and sheds lights to the existing challenges, lessons learned, and best practices of data annotation and machine learning modeling to tackle this problem. Lastly, we present a conclusive solution we took during this research exploration: to detect seasonality, we leveraged Multimodal LLMs (MLMs) which on our in-house benchmark achieved 0.97 top F1 score. Based on our findings, we envision MLMs as a teacher for knowledge distillation, a machine labeler, and a part of the ensembled and tiered seasonality detection system, which can empower ads ranking systems with enriched seasonal information.

Proactive Detection and Calibration of Seasonal Advertisements with Multimodal Large Language Models

TL;DR

Based on the findings, MLMs are envisioned as a teacher for knowledge distillation, a machine labeler, and a part of the ensembled and tiered seasonality detection system, which can empower ads ranking systems with enriched seasonal information.

Abstract

A myriad of factors affect large scale ads delivery systems and influence both user experience and revenue. One such factor is proactive detection and calibration of seasonal advertisements to help with increasing conversion and user satisfaction. In this paper, we present Proactive Detection and Calibration of Seasonal Advertisements (PDCaSA), a research problem that is of interest for the ads ranking and recommendation community, both in the industrial setting as well as in research. Our paper provides detailed guidelines from various angles of this problem tested in, and motivated by a large-scale industrial ads ranking system. We share our findings including the clear statement of the problem and its motivation rooted in real-world systems, evaluation metrics, and sheds lights to the existing challenges, lessons learned, and best practices of data annotation and machine learning modeling to tackle this problem. Lastly, we present a conclusive solution we took during this research exploration: to detect seasonality, we leveraged Multimodal LLMs (MLMs) which on our in-house benchmark achieved 0.97 top F1 score. Based on our findings, we envision MLMs as a teacher for knowledge distillation, a machine labeler, and a part of the ensembled and tiered seasonality detection system, which can empower ads ranking systems with enriched seasonal information.

Paper Structure

This paper contains 24 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Ground Truth Collection
  4. Keyword filtering
  5. Human Labeling
  6. Leveraging Multimodal LLMs
  7. PDCaSa
  8. Task
  9. Detection Models
  10. End-to-end MLM
  11. Late-fused MLM Model
  12. Benchmarking Datasets
  13. Experimental Results
  14. Setup
  15. Single-event performance
  16. ...and 9 more sections

Figures (7)

  • Figure 1: Smoothed Calibration for ads delivered on a major conversion traffic
  • Figure 2: Top secondary keywords observed in seasonal ads during May
  • Figure 3: Human labelling pipeline for seasonal ads
  • Figure 4: a) An overview of model performance, evaluated by F1 score for single-event e2e MLM. b) Multievent results
  • Figure 5: (a) Model performance improves with finetune volume (b) An overview of model performance, evaluated by F1 score for single-event e2e MLM. (c) Incorporating additional modality like image improves the model's performance compared with text input only, validating the importance of multimodality. d) Multievent results
  • ...and 2 more figures