ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising

TL;DR

ContextIQ introduces a multimodal expert-based video retrieval system tailored for contextual advertising. By deploying modality-specific experts for video, audio, captions, and metadata and storing their embeddings in a multimodal database, ContextIQ performs text-to-video retrieval without joint training, using a normalization-threshold-merge aggregation across modalities. It achieves competitive results on standard benchmarks and shows clear gains on an advertising-focused dataset, while enabling long-form content processing, real-time ad serving, and brand safety filters. The work highlights the practical value of modular, interpretable multimodal representations for ad targeting and provides public validation data and tools to support deployment in ad ecosystems.

Abstract

Contextual advertising serves ads that are aligned to the content that the user is viewing. The rapid growth of video content on social platforms and streaming services, along with privacy concerns, has increased the need for contextual advertising. Placing the right ad in the right context creates a seamless and pleasant ad viewing experience, resulting in higher audience engagement and, ultimately, better ad monetization. From a technology standpoint, effective contextual advertising requires a video retrieval system capable of understanding complex video content at a very granular level. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources, limiting their practicality and lacking the key functionalities required for ad ecosystem integration. We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising. ContextIQ utilizes modality-specific experts-video, audio, transcript (captions), and metadata such as objects, actions, emotion, etc.-to create semantically rich video representations. We show that our system, without joint training, achieves better or comparable results to state-of-the-art models and commercial solutions on multiple text-to-video retrieval benchmarks. Our ablation studies highlight the benefits of leveraging multiple modalities for enhanced video retrieval accuracy instead of using a vision-language model alone. Furthermore, we show how video retrieval systems such as ContextIQ can be used for contextual advertising in an ad ecosystem while also addressing concerns related to brand safety and filtering inappropriate content.

Figures (4)

• Figure 1: Multimodal embedding generation pipeline for ContextIQ. Input videos are processed by the metadata extracting module, which uses expert models for extracting objects, actions, places, etc., and converts it into a metadata sentence ($m_i$). Four modality encoders then encode the video frames, audio, caption (transcripts) and metadata in parallel and store the embeddings in a multimodal embeddings DB.
• Figure 2: Multimodal search pipeline for ContextIQ. Text query $t$ is first encoded by different text encoders and the multimodal embedding DB is searched to find similar videos. The aggregation module combines the results obtained from different modalities, and the final results are obtained after applying brand-safety filters utilizing emotion, profanity, and hate speech.
• Figure 3: Validation tool built with Streamlit streamlitStreamlitFaster. Note that the different methods are kept anonymous to remove any bias.
• Figure 4: End-to-End ContextIQ video retrieval system for contextual advertising (ref. Sec. \ref{['sec:contextual_advertising']})